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PREFACE 


RAND  has  undertaken  a  research  effort  focused  on  identifying  and 
analyzing  the  range  of  system  and  subsystem  prototyping  strategies 
available  to  the  Department  of  Defense  (DoD)  and  appropriate  to  the 
acquisition  environment  of  the  late  1980s  and  1990s.  As  part  of  that 
effort,  this  report  examines  the  general  nature  of  prototyping,  devel¬ 
ops  an  analytical  framework  for  thinking  about  prototyping  in 
weapon  system  development,  and  analyzes  past  and  present  prototyp¬ 
ing  programs  within  this  framework. 

This  report  should  be  of  interest  to  officials  and  analysts  both  inside 
and  outside  the  government  concerned  with  improving  the  efficiency 
of  the  weapons  acquisition  process. 

This  work  was  sponsored  by  the  Under  Secretary  of  Defense  for  Ac¬ 
quisition.  It  was  carried  out  in  the  Acquisition  and  Support  Policy 
Program  and  was  performed  under  the  auspices  of  RAND’s  National 
Defense  Research  Institute,  a  federally  funded  research  and  develop¬ 
ment  center  supported  by  the  Office  of  the  Secretary  of  Defense  and 
the  Joint  Staff. 
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SUMMARY 


Confusion  regarding  terms  has  historically  led  to  inconsistencies  in 
acquisition  policy  and  strategies.  One  such  term,  prototyping,  has 
been  used  in  many  different  ways,  and  many  different  prototyping 
strategies  have  been  applied  to  weapon  system  development  pro¬ 
grams.  Thus,  despite  numerous  attempts  to  define  optional  prototyp¬ 
ing  policies,  there  is  little  consensus  on  when,  or  with  what  objectives, 
prototypes  should  be  used. 

This  research  is  an  attempt  to  alleviate  that  problem  by  exploring  the 
nature  and  role  of  prototyping  in  weapon  system  development.  The 
issue  is  of  growing  concern  because  of  increased  interest  expressed  by 
Congress  (and  institutionalized  in  Department  of  Defense  [DoD]  ac¬ 
quisition  regulations)  and  also  because  of  the  changing  acquisition 
environment  (declining  budgets,  fewer  new  program  starts,  and  in¬ 
creasing  system  complexity).  Our  objectives  are  to; 

•  Improve  understanding  of  the  fundamental  nature  and  role  of  pro¬ 
totyping  as  an  acquisition  strategy  in  weapon  system  development. 

•  Examine  the  most  common  prototyping  strategies  and  how  they 
have  changed  over  time. 

•  Investigate  the  relationship  between  prototyping  and  program  out¬ 
comes. 

•  Make  recommendations  on  what  kind  of  policy,  if  any,  DoD  should 
have  regarding  the  use  of  prototyping  in  weapon  system  develop¬ 
ment. 

We  address  these  issues  by  adopting  a  large  sample  empirical  re¬ 
search  approach.  Two  databases  were  developed  to  support  the 
analysis:  a  literature  review  that  included  information  on  287 
systems  and  a  program  manager  survey  of  41  U.S.  weapon  system 
development  programs.  In  both  cases,  conceptual  and  empirical  data 
were  collected.  The  conceptual  information  supported  development  of 
a  conceptual  framework  for  analyzing  prototyping  strategies.  The 
empirical  data  supported  a  thorough  analysis  of  past  prototyping 
experience.  Policy  recommendations  are  derived  from  both  aspects  of 
this  research. 

The  evidence  suggests  that  the  lack  of  a  clear  definition  of  prototyping 
has  contributed  to  a  lack  of  consensus  on  prototyping  policy.  We  have 
defined  a  prototype  as  hardware  or  software  that  allows  hands-on 
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testing  in  a  realistic  environment.  But  the  most  critical  characteristic 
of  a  prototype  is  its  intended  use:  A  prototype  is  intended  to  generate 
information  useful  in  assessing  and  reducing  risks  and  improving  the 
quality  of  acquisition  decision  making.  Hence,  in  scope  and  scale,  it  is 
representative  of  a  concept,  subsystem,  or  production  article  with  po¬ 
tential  utility.  A  protot3rpe  is  not  a  complete  system,  but  rather  it  fo¬ 
cuses  on  selected  areas  of  high  risk  that  are  essential  to  system' suc¬ 
cess.  This  definition  is  broad  because  the  range  of  risks  and  decisions 
that  prototyping  can  address  is  broad.  This  is  reflected  in  the  wide 
spectrum  of  prototyping  experience  documented  in  this  report.  Simi¬ 
larly,  that  experience  suggests  that  prototyping  strategies — the  way 
in  which  prototyping  activities  relate  to  the  larger  acquisition  strat¬ 
egy  for  a  system — have  also  varied  widely  in  their  application.  Given 
this  variability,  it  is  no  wonder  that  a  consensus  on  prototyping  policy 
is  difficult  to  achieve. 

THE  NATURE  AND  ROLE  OF  PROTOTYPING 

The  conceptual  framework  developed  as  part  of  this  research  identi¬ 
fies  three  basic  elements  of  a  prototyping  strategy:  goals,  timing,  and 
integration.  These  were  measured  here  as  purposes  and  objectives  of 
prototyping,  the  phase  in  which  prototyping  occurred,  and  the  level  of 
integration  of  the  prototype  relative  to  a  full  production  article. 
There  is  considerable  program-to-program  variation  across  those  di¬ 
mensions,  indicating  that  prototyping  is  not  a  single  approach  but 
rather  a  complex  set  of  strategies  that  differ  across  many  dimensions. 
These  strategies  vary  widely  in  kind — from  an  emphasis  on  demon¬ 
strating  that  a  technology  is  mature  enough  to  be  incorporated  into  a 
weapon  system  to  demonstrating  that  a  particular  combination  of 
system  design  and  technology  meets  operational  requirements. 

A  few  prototyping  strategies  appear  to  be  most  common.  For  in¬ 
stance,  prototyping  activities  with  goals  of  technology  demonstration 
usually  occur  earlier  in  a  progfram  and  are  performed  at  the  partial 
system  level,  with  just  enough  integration  to  demonstrate  the  rele¬ 
vant  new  functions.  On  the  other  hand,  goals  that  focus  more  on  as¬ 
sessing  the  performance  of  a  system  design  in  meeting  an  operational 
need  tend  to  be  associated  with  prototyping  activities  performed  at 
the  full  system  level  (higher  degree  of  integration)  later  in  the  acqui¬ 
sition  cycle.  Additionally,  achievement  of  some  goals  often  precludes 
achievement  of  others. 

There  are  few  identifiable  differences  in  prototyping  strategies  and 
their  application  between  services  or  across  weapon  types.  Appar¬ 
ently,  differences  in  the  technical  characteristics  of  systems,  man- 
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agement  style,  and  institutional  structure  are  not  major  drivers  of  ob¬ 
served  differences  in  prototyping  strategies.  The  application  of  a  par¬ 
ticular  prototyping  approach  was  more  a  function  of  the  purposes  and 
objectives  of  the  prototyping  activities,  which  in  turn  was  a  function 
of  the  type  of  risks  and  uncertainties  that  exist  in  the  program. 

Detailed  progprammatic  characteristics,  such  as  contract  type,  re¬ 
quirements  specification,  number  of  prototypes,  decision  layers,  and 
amount  of  documentation,  are  loosely  associated  with  the  basic  strat¬ 
egy  elements.  There  are  myriad  detailed  programmatic  characteris¬ 
tics  that  both  contribute  to  defining  a  particular  strategy  and  account 
for  the  considerable  variation  in  the  specifics  of  prototyping  strate¬ 
gies.  For  the  most  part,  these  detailed  programmatic  characteristics 
flow  from  the  basic  elements  of  prototyping  strategy:  the  goals,  tim¬ 
ing,  and  integration  level  of  the  prototyping  activities. 

Macro-level  strategies  have  been  fairly  constant  over  time,  though 
some  factors  may  have  changed  at  the  detailed  level.  There  are  some 
indications  of  minor  change;  focus  on  more  mature  technology  and 
demonstrations  of  performance  achievement  rather  than  technical 
feasibility.  But  the  evidence  suggests  that  prototyping  strategies  of 
the  past  are  for  the  most  part  just  as  common  today,  despite  changes 
in  the  acquisition  environment. 

Prototyping  strategies  do  not  seem  sensitive  to  factors  that  we  might 
expect  to  affect  them.  The  number  of  prototypes  fabricated  and  tested 
does  not  seem  to  vary  across  the  different  combinations  of  purposes, 
objectives,  integration  levels,  or  program  phases.  Similarly,  macro¬ 
level  prototyping  strategies  appear  to  be  only  slightly  related  to  the 
cost  and  schedule  aspects  of  a  program. 

PROTOTYPING  AND  PROGRAM  OUTCOMES 

The  effect  of  protot3q)ing  on  program  cost,  schedule,  and  performance 
outcomes  is  ambiguous.  We  might  expect  that  the  information  gener¬ 
ated  through  prototyping  activities  would  reduce  program  risk  as 
compared  to  nonprototyping  prog^rams.  Assuming  incorporation  of 
the  information  into  progpram  plans  and  baselines,  prototyping  pro¬ 
gram  outcomes  should  be  relatively  closer  to  baseline  estimates. 
However,  there  are  few  significant  differences  between  prototyping 
and  nonprototyping  programs  with  respect  to  cost  growth,  total  actual 
progpram  duration,  or  schedule  slip.  Similarly,  it  is  not  possible  to 
generalize  about  the  effects  of  certain  types  of  prototyping  strategies 
relative  to  others.  We  do  not  know  enough  about  the  mechanisms 
through  which  prototyping  affects  program  outcomes  to  be  able  to  es- 


tabhsh  a  relationship  between  prototyping  strategies  or  program¬ 
matic  characteristics  and  particular  outcomes,  nor  do  we  know  how  to 
distinguish  those  effects  from  the  myriad  of  other  factors  that  may 
influence  program  outcomes. 


POUCY  IMPLICATIONS 

The  research  questions  explored  in  this  analysis  fall  under  the  um¬ 
brella  policy  goal  of  establishing  criteria  for  prototyping:  deciding 
whether  or  not  to  prototype,  and  if  so,  how.  This  analysis  suggests 
that  attempts  to  define  such  a  policy  at  any  but  the  broadest  levels 
should  be  avoided.  Program-specific  characteristics  and  the  charac¬ 
teristics  of  the  acquisition  environment  vary  so  widely  that  no  generic 
criteria  are  apparent  to  determine  whether  or  not  to  prototype  or 
what  kind  of  prototyping  strategy  to  pursue.  Thus,  it  is  not  possible 
or  even  desirable  to  develop  a  set  of  firm  decision  rules.  Nonetheless 
based  on  the  research  reported  here,  we  can  say  something  about 
what  broad  acquisition  policy  with  respect  to  prototyping  should  be. 

The  basic  policy  ^ideline  might  require  program  managers  to  assess 
the  extent  to  which  prototyping  is  both  useful  and  appropriate  in  a 
particular  case.  In  essence,  this  requires  an  evaluation  of  the  relative 
costs  and  benefits  of  prototyping  for  that  application.  In  terms  of 
dollars,  this  might  be  stated  as  weighing  the  cost  consequences  of 
proceeding  into  a  subsequent  acquisition  phase  with  a  poor  design 
against  the  cost  of  the  prototyping  strategy.  Of  course,  determining 
the  cost  consequences  of  proceeding  with  inadequate  information 
(e.g.,  cost  nsk)  is  not  a  trivial  exercise.  Similarly,  the  fundamental 
consideration  might  be  stated  as  a  trade-off  between  the  net  benefits 
of  prototyping  and  the  type  and  magnitude  of  risk  in  a  development 
effort.  Hence,  prototyping  policy  might  require  evaluating  and  bal¬ 
ancing  several  factors:  the  relative  technological  advance  in  a  system 
unrertainty  regarding  the  utility  of  a  new  concept,  the  maturity  of  the' 
technology  being  incorporated,  and  cost  and  schedule  constraints. 
Unfortunately,  there  is  no  way  to  consistently  measure  those  dimen¬ 
sions.  Further,  even  if  we  could  accurately  measure  the  relative 
technological  maturity  of  a  system  design,  for  instance,  we  still  do  not 
know  how  much  weight  to  give  any  single  factor  in  the  overall  deci¬ 
sion  calculus.  In  the  end,  there  is  no  substitute  for  informed  judg¬ 
ment  made  by  experienced  managers  and  engineers. 
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1.  INTRODUCTION 


This  report  explores  the  nature  of  prototyping,  analyzes  the  role  it 
plays  in  weapon  system  development,  and  extracts  lessons  for  acqui¬ 
sition  policy  making.  Prototyping  can  be  simplistically  defined  as  the 
fabrication  and  testing  of  hardware  or  software  at  some  point  in  the 
acquisition  cycle  prior  to  commitment  of  full  rate  production.  (This 
research  adds  precision  to  that  definition  of  prototyping.) 

The  use  of  protot3rping  in  weapon  system  development  has  been  cycli¬ 
cal.  Prototyping  of  aircraft  engine-airframe  combinations  was  the 
customary  pattern  of  aircraft  development  before  1940  and  was  fairly 
common  into  the  1950s.  With  the  advent  of  the  “total  system  concept” 
in  the  early  1950s,  the  use  of  prototyping  in  U.S.  weapon  system  de¬ 
velopment  declined.  The  pattern  again  reversed  in  the  late  1960s 
when  (then)  Deputy  Secretary  of  Defense  Packard  instituted  a  “fly- 
before-buy”  acquisition  strategy,  which  emphasized  hardware  demon¬ 
strations  before  production  start.  However,  the  military  services 
resisted  the  policy,  and  it  was  not  fully  accepted  and  applied. 

In  the  mid-1980s,  U.S.  interest  in  the  use  of  prototyping  in  weapon 
system  development  again  surged:  The  1986  Packard  Commission 
report  emphasized  that  prototyping  before  commitment  to  full-scale 
development  would  enhance  source  selection,  provide  early  demon¬ 
strations  of  the  feasibility  and  operational  utility  of  new  technologies, 
and  improve  initial  cost  estimates  of  new  systems.*  Congress  also 
began  to  encourage  prototyping  of  weapon  systems,  and  in  1987  pro¬ 
totyping  formally  became  a  part  of  Department  of  Defense  (DoD)  ac¬ 
quisition  regulations.^  Nonetheless,  basic  questions  concerning  the 
nature  of  prototyping,  under  what  conditions  one  should  prototype, 
and  the  benefits  of  prototyping  remain  unanswered. 

The  changing  acquisition  environment  may  also  imply  changes  for 
prototyping  strategy.  In  the  past,  fairly  complete  systems  could  be 
prototyped  at  a  reasonable  cost.  But  as  weapons  become  more  com¬ 
plex,  prototyping  full-scale,  fully  integrated  systems  may  no  longer  be 
feasible  or  even  beneficial.  The  nature  of  the  technical  challenge  has 
also  changed;  in  aircraft,  increased  emphasis  is  being  placed  on  inte- 


*A  Quest  for  Excellence,  Final  Report  to  the  President  by  the  President’s  Blue 
Ribbon  Commission  on  Defense  Management,  June  1986. 

^DoD  Instruction  5000.2,  “Defense  Acquisition  Program  Procedures,'  September  14, 
1987. 
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grated  electronics,  economical  supercruise,  agility,  and  low  observable 
characteristics;  in  tanks,  emphasis  is  on  armor,  gun  aiming,  and  ar¬ 
mor  penetration.  The  simultaneous  introduction  of  these  new  tech¬ 
nologies  and  changes  in  emphasis  among  the  various  performance 
characteristics  (e.g.,  emphasizing  stealth)  seem  likely  to  increase 
technical  integration  risks  and  suggest  that  there  might  be  corre¬ 
sponding  changes  in  the  appropriateness  or  application  of  prototyping 
strategies.  Additionally,  the  projected  decline  in  defense  budgets  and 
the  associated  decrease  in  new  major  system  starts  suggest  a  new  role 
for  prototyping  in  the  future;  as  a  way  to  keep  design  teams  together 
and  continue  to  advance  the  state  of  the  art  in  weapon  system  tech¬ 
nologies.^  To  devise  protot3T)ing  strategies  that  will  meet  these  new 
challenges,  we  must  first  understand  the  nature,  role,  and  impact  of 
current  prototyping  efforts. 

OBJECTIVES 

Whether  or  not  to  prototype — and,  if  so,  how — is  an  important  deci¬ 
sion  in  the  acquisition  process,  with  implications  for  program  cost, 
schedule,  and  performance  outcomes.  But  that  issue  cannot  be  re¬ 
solved — or  even  fully  addressed — until  more  basic  questions  are  an¬ 
swered.  Confusion  and  ambiguity  in  the  use  of  the  term  prototype  has 
been  a  major  contributor  to  the  lack  of  consensus  for  the  use  of  proto¬ 
types  in  weapon  system  development.  What  is  a  prototype?  Strictly 
the  earliest,  nonoperational,  functionally  representative  test  article  in 
a  development  program?  Or  any  test  article  built  prior  to  production? 
Likewise,  what  distinguishes  prototyping  from  other  acquisition 
strategies?  Is  it  one  strategy,  or  several?  And  if  several  prototyping 
strategies  exist,  how  are  they  different?  What  are  their  individual 
benefits  and  limitations? 

This  research  begins  to  address  these  questions,  and  also  begins  to 
explore  the  policy  implications  of  the  answers  to  these  questions.  We 
have  four  objectives: 

•  Improve  understanding  of  the  fundamental  nature  and  role  of  pro¬ 
totyping  as  an  acquisition  strategy  in  weapon  system  development. 


^This  proposal  has  appeared  in  several  forms  from  several  sources  recently:  a  1990 
DSB  Summer  Study;  ‘rollover”  policy  as  proposed  by  Representative  Les  Aspin;  Paul 
H.  Richanbach  et  al.,  The  Future  of  Military  R&D:  Towards  a  Flexible  Acquisition 
Strategy,  Institute  for  Defense  Analyses,  July  1990,  P-2444;  U.S.  Congress, 
Restructuring  Defense:  Transitioning  the  Defense  Industrial  Base,  Office  of  Technology 
Assessment,  July  1991. 
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•  Examine  the  most  common  prototyping  strategies  and  how  they 
have  changed  over  time. 

•  Investigate  the  relationship  between  prototyping  and  program  out¬ 
comes. 

•  Make  recommendations  on  what  kind  of  policy,  if  any,  DoD  should 
have  regarding  the  use  of  prototyping  in  weapon  system  develop¬ 
ment. 

The  first  objective  constitutes  the  major  research  focus.  It  is  oriented 
toward  developing  an  operational  definition  of  prototype  and  prototyp¬ 
ing  strategy.  This  includes  distinguishing  prototypes  from  other 
hardware  used  for  testing,  and  differentiating  prototyping  strategies 
from  other  acquisition  strategies  (e.g.,  conventional  development- 
production  or  concurrent  development-production  approaches).  Gen¬ 
erally,  improved  understanding  of  prototyping  as  an  acquisition 
strategy  can  be  attained  only  by  answering  these  initial  questions. 
Development  of  a  conceptual  framework  and  characterization  of  past 
prototyping  activities  is  an  integral  part  of  the  research  addressing 
those  questions.  This  objective  attempts  to  crystalize  the  concept  of 
prototyping  and  its  place  in  weapon  systems  development,  thus 
enabling  more  consistent  and  informed  policy  making  regarding  its 
use. 

The  second  objective  takes  the  research  past  the  development  of  a 
conceptual  framework  and  begins  to  address  policy  issues.  We  are  in¬ 
terested  here  in  identifying  the  more  common  prototyping  strategies, 
if  any.  We  also  begin  to  address  the  issue  of  the  extent  to  which  pro¬ 
totyping  strategies  need  to  be  changed  to  reflect  changes  in  the  ac¬ 
quisition  environment.  Technical,  political,  and  economic  changes  in 
the  acquisition  environment  (e.g.,  subsystem  integration,  increased 
program  oversight,  and  declining  R&D  budgets)  suggest  that  past 
prototyping  strategies  and  associated  benefits  may  not  be  entirely  ap¬ 
propriate  in  the  current  environment.  Identifying  how  prototyping 
strategies  have  changed  over  time  is  a  first  step  toward  assessing 
whether  past  prototyping  strategies  are  relevant  to  the  new  environ¬ 
ment. 

The  third  objective  attempts  to  provide  some  insight  into  the  relation¬ 
ship  of  prototyping  and  program  outcomes.  Previous  research  has 
demonstrated  that  prototyping  can  be  a  valuable  approach  to 
weapons  development;  at  least  for  some  programs,  it  has  reduced  cost 
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growth,  schedule  slippage,  and  performance  shortfalls^  It  has  also 
enabled  decision  makers  to  make  better-informed  decisions.  Though 
this  study  does  not  assess  the  value  of  prototyping  per  se,  it  addresses 
related  questions:  Does  prototyping  add  time  to  program  schedules? 
Is  prototyping  associated  with  lower  cost  growth  and  schedule  slip 
outcomes?  What  are  the  mechanisms  for  such  an  effect?  Prototyping 
policy  should  be  based  on  knowledge  of  the  relationship  between  pro¬ 
totyping  and  program  outcomes. 

The  fourth  objective,  making  policy  recommendations,  describes  what 
DoD  prototyping  policy  might  look  like,  how  it  relates  to  other  acqui¬ 
sition  policies,  and  how  it  might  be  implemented.  The  ultimate  goal 
of  such  research  is  to  produce  criteria  that  would  define  when  and 
how  prototyping  should  be  used  in  weapons  development — a  particu¬ 
larly  important  task  now,  given  the  changing  nature  of  military  sys¬ 
tems,  inventories,  and  shifts  in  acquisition-spending  patterns.  We 
are  attempting  to  produce  a  tool  that  will  help  decision  makers  de¬ 
termine  whether  or  not  a  prototyping  strategy  is  worthwhile  in  a  spe¬ 
cific  instance.  While  this  research  stops  short  of  providing  actual  cri¬ 
teria  or  specifying  a  detailed  poli<^,  it  does  offer  a  conceptual  outline 
of  how  prototyping  policy  should  evolve. 

GENERAL  APPROACH 

Prototyping  is  only  interesting  as  a  policy  issue  to  the  extent  that  pro¬ 
totyping  is  likely  to  significantly  influence  program  outcomes.  Thus  a 
comparison  of  prototyping  and  nonprototyping  programs  is  required. 
While  other  research  has  performed  such  comparisons  using  a  case 
study  approach,  there  are  few  examples  of  this  type  of  analysis  using 
a  larger,  broader-based  sample. 

This  research  uses  a  large  sample  size  approach  in  an  attempt  to  bet¬ 
ter  understand  the  nature  of  prototyping  and  its  role  in  weapon  sys¬ 
tem  development.  In  this  approach,  some  detail  is  sacrificed  to  enable 
a  study  of  broader  scope.  A  limited  number  of  variables,  specified  in 
advance,  is  collected  for  a  large  number  of  programs,  and  subjected  to 
statistical  analysis.  While  the  broader  scope  and  macro-level  per¬ 
spective  of  this  large  sample  approach  is  more  conducive  of  policy 
formulation,  it  is  unsatisfying  in  the  sense  that  cause-effect  relation- 


^See  Edmund  Dews  et  al.,  Acquisition  Policy  Effectiveness:  Department  of  Defense 
Experience  in  the  1970s,  RAND,  R-2S16-DR&E,  O^ber  1980;  Karen  W.  Tyson  et  al., 
Acquiring  Major  Systems:  Cost  and  Schedule  Trends  and  Acquisition  Initiative 
Effectiveness,  Institute  for  Defense  Analyses,  March  1989,  P-2201. 
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ships  cannot  be  determined.  The  factors  affecting  the  type  of  proto- 
typing  strategy  chosen,  and  the  effect  of  that  strategy  on  program 
outcomes,  cannot  be  definitively  known. 

Case  studies,  on  the  other  hand,  provide  considerable  detail  regarding 
the  role  of  prototyping  in  a  few  specific  programs.®  Alone,  this  ap¬ 
proach  is  unsatisfying  in  the  sense  that  the  results  are  not  necessarily 
generally  applicable;  it  is  difficult  to  formulate  general  policy  from  a 
few  case  studies.  However,  case  studies  do  allow  identification  of  the 
factors  affecting  the  type  of  prototyping  strategy  used  and  help  us 
better  understand  the  relative  success  of  that  strategy. 

While  neither  approach  alone  appears  wholly  satisfying,  together 
they  provide  a  firm  basis  for  policy  making.  The  large-sample  empiri¬ 
cal  approach  adopted  here  has  not  often  been  used  with  respect  to 
prototyping.  We  can  answer  questions  regarding  what  a  prototype  is, 
how  it  has  been  used  in  the  past,  and  how  the  components  of  proto¬ 
typing  strategies  relate  to  each  other.  We  cannot  answer  questions 
about  the  appropriateness  of  given  prototyping  application,  or  its  spe¬ 
cific  effect  on  a  program.  Hence,  a  case  study  approach  is  also  needed 
to  accurately  answer  those  questions.  We  do  include  a  limited  com¬ 
parison  between  prototyping  and  nonprototyping  programs  with  re¬ 
spect  to  certain  measurable  program  outcomes  (cost  growth,  program 
duration,  schedule  slip)  in  order  to  gain  some  macro-level  insight  into 
the  effect  of  prototyping  on  program  outcomes,  but  these  results  can¬ 
not  be  definitive.  Since  the  research  is  not  complete  in  itself,  only 
limited,  macro-level  policy  implications  can  be  drawn  from  the  analy¬ 
sis. 

Much  of  the  information  used  here  comes  from  various  sources  in 
publicly  available  literature.  Many  different  journals  and  reports 
were  reviewed  in  an  attempt  to  compile  a  list  of  prototyping  activities 
since  1960,  both  foreign  and  domestic,  and  synthesize  the  wide  range 
of  concepts  and  philosophy  surrounding  the  use  of  prototypes  in  ac¬ 
quisition.  Because  much  of  the  literature  is  ambiguous,  a  formal  sur¬ 
vey  of  government  program  managers  was  also  conducted.  This  sur¬ 
vey  provided  more  detailed  and  better  quality  information  on  the 
characteristics  of  prototyping  strategies  that  both  supplemented  and 
substantiated  the  data  derived  from  the  literature.  Additionally,  a 
database  containing  cost  and  schedule  information  on  a  large  number 


®See  for  example  G.  K.  Smith  et  al..  The  Use  of  Prototypes  in  Weapon  System 
Development,  RAND,  March  1981,  R-2345-AF;  and  Mark  A.  Lorell,  The  Use  of 
Prototypes  in  Selected  Foreign  Fighter  Aircraft  Development  Programs,  RAND,  R-3687- 
P&L,  September  1989. 
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of  programs,  both  prototyping  and  nonprototyping,  was  used  to  ex¬ 
plore  the  relationship  of  prototyping  to  program  outcomes.^ 

REPORT  ORGANIZATION 

The  remainder  of  this  report  presents  the  results  of  the  analysis.  Sec¬ 
tion  2  defines  prototype  and  prototyping,  discusses  several  related 
concepts,  and  develops  the  conceptual  framework  to  describe  and  ana¬ 
lyze  prototyping  strategies.  Section  3  outlines  the  study  design  and 
data  used  to  analyze  prototyping  experience.  Both  published  and 
survey  data  are  included.  Section  4  presents  the  basic  findings  relat¬ 
ing  to  general  prototyping  strategies  and  analyzes  variations  in  their 
basic  elements  and  trends  over  time.  Section  5  attempts  to  relate 
prototyping  to  cost  and  schedule  outcomes,  including  some  compari¬ 
son  with  nonprototyping  programs.  Section  6  draws  policy  implica¬ 
tions  from  the  lessons  learned  in  this  analysis  and  presents  further 
observations  on  prototyping  policy.  While  this  section  stops  short  of 
providing  specific  guidelines  for  the  use  of  prototyping  in  weapon  sys¬ 
tem  development,  it  does  describe  the  characteristics  of  a  robust  pro¬ 
totyping  policy. 

Three  appendices  contain  supplemental  information.  For  readers  in¬ 
terested  in  the  mix  of  programs  used  in  this  research.  Appendix  A 
provides  the  lists  of  programs  included  in  the  analysis,  both  the  liter¬ 
ature  review  and  survey  databases.  Appendix  B  contains  more  de¬ 
tailed  tables  and  graphs,  which  document  some  additional  results  of 
the  program  manager  survey  and  the  literature  review  These  data 
supplement  the  data  in  Section  4  on  the  characteristics  of  prototyping 
strategies.  Appendix  C  reprints  the  program  manager  worksheet,  the 
survey  instrument  used  to  collect  information  from  program  man¬ 
agers. 


^This  database  is  derived  from  Selected  Acquisition  Reports  and  is  based  on  ongoing 
research  by  the  author. 


2.  A  CONCEPTUAL  FRAMEWORK  FOR 
PROTOTYPING 


Consistent  policy  formulation  regarding  the  use  of  prototyping  in 
weapon  system  development  requires  some  agreement  on  what  proto¬ 
typing  means.  A  cohesive  framework  that  captures  the  variability  of 
prototyping  concepts  and  relates  that  variability  to  the  characteristics 
of  prototypes,  the  development  process,  and  the  acquisition  environ¬ 
ment  has  been  lacking.  This  section  is  an  effort  to  construct  such  a 
framework.  The  goal  is  to  provide  a  structure  that  enables  improved 
decisions  on  the  use  of  prototyping  in  development  programs. 

This  section  begins  by  defining  prototype  and  prototyping,  then  pro¬ 
ceeds  to  develop  the  framework  that  will  be  used  throughout  the 
analysis.  The  framework  is  based  on  an  extensive  literature  review, 
discussions  with  knowledgeable  industry  and  government  personnel, 
and  the  analysis  presented  here.  It  identifies  the  most  important 
characteristics  of  prototyping  programs,  and,  combining  them,  it 
divides  prototypes  into  several  categories.  Like  any  attempt  to  sim¬ 
plify  a  large  and  highly  variable  body  of  data,  this  taxonomy  is  some¬ 
what  imprecise.  Yet  each  category  defines  a  g^roup  of  prototyping 
strategies  that  offers  unique  benefits  and  brings  unique  limitations. 
By  using  this  framework  to  analyze  past  development  programs,  we 
can  learn  how  to  use  prototyping  most  effectively.  For  example,  one 
class  of  prototype  may  be  helpful  for  a  given  type  of  program  while 
another  class  would  be  not  only  less  useful,  but  even  counterproduc¬ 
tive. 

DEFINING  TERMS 

Before  we  can  distinguish  prototypes  from  one  another,  we  must  first 
distinguish  them  from  everything  else.  The  most  inclusive  definition 
of  a  prototype  is  “hardware  used  for  testing.”  This  definition  is  not 
useful  for  acquisition  policy,  however,  because  it  fails  to  distinguish 
prototypes  from  the  hardware  test  articles  often  built  during  full  scale 
development  (FSD)  or  production.  Neither  advocates  nor  critics  of 
prototyping  mean  to  include  these  test  articles  in  the  debate. 

What,  then,  defines  a  prototype?  One  source  describes  it  as  a  tool 
for  reducing  technological  uncertainty,  particularly  with  respect  to 
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cost,  schedule,  performance,  tactics,  and  integrated  logistics  support.^ 
For  aircraft,  this  definition  translates  to  an  engine-airframe 
combination  that  approximates  the  main  features  of  a  proposed  oper¬ 
ational  aircraft.^  An  alternative  definition  holds  that  a  prototype  is  “a 
vehicle  or  component  the  primary  purpose  of  which  is  to  test  a  design 
concept  and  obtain  the  information  necessary  for  making  sound 
decisions.  . . 

The  benefits  commonly  ascribed  to  prototyping  imply  support  for  both 
positions.  The  benefits  listed  in  Table  2.1  (distilled  from  various 
sources)  have  a  common  theme:  Prototypes  generate  information  in 
order  to  reduce  risk. 

Since  both  prototyping  and  FSD  test  articles  provide  information  to 
reduce  risk,  however,  an  additional  dimension  is  needed  to  distin¬ 
guish  between  them.  That  dimension  is  the  intended  use  of  the  in¬ 
formation.  As  one  definition  explains,  a  prototype  is  built  in  the 


Table  2.1 

Perceived  Benefits  of  Prototyping 


Frequently  Mentioned  Benefits 

•  Reduces  technological  risk  and  uncertainty 

■  Enables  better  quality  decisions  regarding  trade-offs  between  cost,  schedule,  and 
performance 

■  Permits  changes  to  be  incorporated  early  in  the  program 

•  Identifies  system  interfaces  and  key  technical  problems 

•  Permits  improved  estimates  of  cost,  schedule,  and  performance 

•  Increases  design  options 

•  Permits  earlier  testing  (development  and  operational) 

■  Provides  a  hedge  against  threat  uncertainty 

Other  Possible  Benefits 

•  Cost  effectiveness  (if  funding  is  austere) 

•  Improved  visibility  of  logistics  support  and  life  cycle  costs 

■  Improved  response  time  to  changes  in  threat 

•  Lower  tooling  and  retooling  costs 

•  Improved  government  contract  negotiating  position 

•  Less  government  oversight  in  competitive  environment 

■  Sequential  development  and  testing 


'Robert  Perry,  A.  Prototype  Strategy  for  Aircraft  Development,  RAND,  RM-5597-1- 
PR,  July  1972. 

2lbid. 

^B.  H.  Klein,  T.  K.  GIcnnan,  and  G.  H.  Shubert,  The  Role  of  Prototypes  in 
Development,  RAND,  RM-3467-PR,  February  1963. 
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expectation  of  change/  The  information  that  it  generates  affects  de¬ 
cisions  on  source  selection;  cost,  schedule,  and  performance  trade-offs; 
operational  utility;  and  the  major  program  phase  milestones.  This 
implies  that  the  results  of  prototyping  will  be  included  in  a  major 
technical  or  programmatic  decision  prior  to  the  production  decision. 
One  possible  decision  is  program  termination.  Such  flexibility  in  de¬ 
cision  making  is  not  often  found  during  testing  of  FSD  articles. 

Thus  we  have  a  fairly  robust  definition;  a  prototype  is  a  product 
(hardware  and  lor  software)  that  allows  hands-on  testing  in  a  realistic 
environment.  In  scope  and  scale,  it  represents  a  concept,  subsystem,  or 
production  article  with  potential  utility.  It  is  built  to  improve  the 
quality  of  decisions,  not  merely  to  demonstrate  satisfaction  of  contract 
specifications.  The  expectation  (and  acceptance)  of  change  allows  the 
information  generated  by  the  prototype  to  be  included  in  major  deci¬ 
sions  affecting  both  risk  management  and  the  outcome  of  the  pro¬ 
gram. 

A  prototype  is  not  a  complete  system  in  the  sense  of  being  deployable 
to  operational  forces.  Rather,  a  prototype  focuses  on  selected  areas  of 
high  risk  that  are  essential  to  system  success.  Additionally,  the  de¬ 
sire  to  minimize  the  cost  and  time  of  prototyping  suggests  that  limita¬ 
tions  in  nonessential  areas  are  acceptable.  For  example,  a  prototype 
aircraft  may  not  be  designed  to  sustain  the  higher  level  of  lifetime 
flight  hours  as  an  FSD  or  production  article. 

So  the  distinction  between  prototypes  and  other  test  articles  lies  in 
the  purpose  of  the  device,  the  information  gained  through  testing,  and 
how  this  information  is  meant  to  be  used  in  the  decision  process. 
Prototyping,  the  strategy  of  using  prototypes  in  the  acquisition  pro¬ 
cess,  can  be  defined  in  the  same  way.  It  is  the  strategy  of  using  test 
articles  to  generate  knowledge  intended  to  reduce  identified  risks  and 
uncertainties. 

Prototyping  is  an  alternative  to  other  acquisition  strategies,  such  as  a 
conventional  development-production  approach,  and  can  supplement 
other  kinds  of  analysis  (e.g.,  wind  tunnel  tests).  A  conventional  or 
concurrent  development-production  approach  differs  from  a  prototyp¬ 
ing  approach  in  terms  of  the  relative  timing  of  test  information  avail¬ 
ability  and  programmatic  decision  making,  and  the  nature  of  those 
decisions.  In  conventional  development-production  programs,  the  in¬ 
tent  to  transition  to  production  as  rapidly  as  possible  is  inherent  in 
program  planning  at  the  time  the  development  contract  is  awarded. 


^Perry,  July  1972,  p.  6. 
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Test  articles  built  during  development  have  the  sole  purpose  of 
demonstrating  achievement  of  contract  specifications.  In  contrast,  a 
much  wider  range  of  risks  are  addressed  in  prototype  testing,  gener¬ 
ating  information  that  informs  a  wider  range  of  decisions,  including 
the  decision  to  continue  development  or  terminate  the  program. 
While  there  is  some  overlap,  the  basic  distinction  concerns  the  intent 
to  use  prototjrpe  test  information  for  a  broader  set  of  more  fundamen¬ 
tal  decisions. 

CLASSIFYING  PROTOTYPING  STRATEGIES 

The  next  step  is  to  construct  a  taxonomy,  a  scheme  to  classify  the  var¬ 
ious  kinds  of  prototypes.  That  effort  begins  by  identifying  the  impor¬ 
tant  characteristics  that  have  been  used  to  classify  prototypes.  Like 
attempts  to  define  prototyping,  most  previous  efforts  to  classify  proto¬ 
types  have  focused  on  their  use. 

Many  classification  schemes  are  possible  based  on  the  many  different 
uses  and  characteristics  of  prototyping.  For  instance,  an  advanced 
prototype  can  be  used  to  “verify  and  reduce  the  technology”  of  hard¬ 
ware,  evaluate  operational  concepts,  and  provide  alternative  choices, 
while  a  production  prototype  reflects  full  production  design  details 
and  is  built  using  hard  tooling.®  Critical  subsystem  prototyping  in¬ 
volves  development  of  critical  components  and  subsystems  indepen¬ 
dently  from  full  system  prototyping.®  Categories  of  prototypes  are 
often  associated  with  the  development  stage  in  which  they  occur,  re¬ 
flecting  both  degrees  of  system  or  technological  maturity  and  a  spec¬ 
trum  of  program  uncertainties.  Notably,  many  of  these  categories 
correspond  well  with  the  intent  of  a  DoD  regulation.^  Yet  another 
possibility  is  to  group  prototypes  by  goal.  These  might  fall  into  three 
separate  categories;  increasing  development  efficiency,  improving  the 
quality  of  decisions,  and  hedging  against  uncertainties.  Achievement 
of  one  of  these  goals  is  not  necessarily  exclusive  of  other  goals.®  It  is 
interesting  that  these  goals  are  a  restatement  of  those  commonly  at¬ 
tributed  to  prototyping  (see  Table  2.1). 


®David  Packard,  ‘Improving  R&D  Management  Through  Prototyping,*  Defense 
Management  Journal,  July  1972,  p.  5. 

®Michael  A.  Pearce,  Prototyping:  A  Strategy  for  the  Acquisition  of  Naval  Aircraft, 
Defense  Systems  Management  College,  10  November  1976. 

^MIL-STD  250A.  This  regulation  defines  exploratory,  advanced  development, 
engineering  development,  preproduction,  and  production  models. 

*0.  K.  Smith  et  al.,  The  Use  of  Prototypes  in  Weapon  System  Development,  RAND, 
R-2345-AF,  March  1981. 
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Clearly,  prototypes  can  be  classified  across  a  number  of  dimensions, 

many  of  which  are  also  program  characteristics.  A  representative 

list — ^by  no  means  exhaustive — might  include  the  following  dimen¬ 
sions: 

•  Type  of  risk:  technical — ^technological  feasibility;  target — “reducing 
a  military  need  to  cost,  schedule,  and  performance  goals”;  inter¬ 
nal — program  management;  and  process — ^funding  and  approval.® 

•  Degree  of  risk:  magnitude,  seriousness,  and  importance  of  the  var¬ 
ious  types  of  risk. 

•  Level  of  system  integration:  indicates  what  portion  of  the  key  sub¬ 
systems  are  in  the  prototype  in  their  final  functional  form. 

•  Program  phase  in  which  prototyping  activities  occur:  reflects  type 
and  timing  of  information  availability. 

•  Austerity  of  funding  and  management:  how  narrowly  focused  the 
effort  is,  and  how  much  flexibility  the  sponsor  allows  the  contractor 
in  making  trade-offs. 

•  Degree  of  production  expectation:  intent  to  produce  and  deploy 
system. 

•  Degree  to  which  technical,  performance,  and  mission  requirements 
and  objectives  are  specified:  whether  point  goals  are  written  into 
the  contract  or  whether  “best  effort”  or  performance  range  is  ac¬ 
ceptable. 

•  Form  with  respect  to  final  design  and  configuration:  indicates  the 
representativeness  of  the  prototype. 

•  Tooling:  soft  vs  hard. 

•  Decision  affected:  can  be  any  technical  or  milestone  decision  prior 
to  full-rate  production. 

•  Budget  category:  basic  research  (6.1),  exploratory  development 
(6.2),  advanced  development  (6.3),  and  engineering  development 
(6.4). 

•  Degree  of  management  flexibility:  reflecting  ability  to  make  cost, 
schedule,  and  performance  trade-off  decisions  at  the  program  office 
and  contractor  level. 

•  Documentation  and  reporting  requirements:  reflecting  need  for 
tracking,  coordination,  and  accountability. 


®The8c  risk  categories  are  defined  in  William  E.  Thompson,  HI,  “Risk  Implications 
for  Cost  Growth  in  Weapon  System  Acquisition,*  Concepts — The  Journal  of  Defense 
System  Acquisition  Management,  Spring  1982,  Vol.  5,  No.  2. 
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•  Quality  of  labor:  relative  to  the  production  phase. 

There  are  strong  relationships  among  these  dimensions.  The  type 
and  degree  of  risk,  for  example,  affect  the  phase  in  which  the  proto¬ 
type  occurs  and  the  subsequent  decisions.  Austerity  is  reflected  in 
funding  and  management  flexibility.  Production  expectations  and 
tooling  are  also  closely  linked. 

GOAL:  A  NEW  DIMENSION 

Our  objective  is  to  create  a  framework  that  includes  most  of  the  criti¬ 
cal  dimensions  that  define  a  prototyping  strategy,  yet  remains  simple 
enough  to  be  useful  in  decision  making.  We  suggest  three  key  ele¬ 
ments;  timing,  level  of  system  integration,  and  goal.  From  the  per¬ 
spective  of  the  framework,  the  type  and  degree  of  risk  in  large  part 
determine  the  choice  of  timing,  integration  level,  and  goals,  while  the 
various  other  programmatic  characteristics  listed  above  (e.g.,  auster¬ 
ity,  tooling,  production  expectation)  flow  from  those  choices.  The  first 
two  dimensions  are  simple  and  widely  used.  Timing  means  the  phase 
in  which  prototyping  occurs,  and  it  is  related  to  the  level  of  system  or 
technological  maturity.  The  level  of  system  integration  means  the  ex¬ 
tent  to  which  the  prototype  represents  a  production  unit  in  scope  and 
scale,  and  includes  all  necessary  subsystems  for  operational  deploy¬ 
ment.  The  third  element  of  our  framework,  goal,  is  not  well  repre¬ 
sented  in  the  literature  on  classifying  prototypes.  Because  it  requires 
more  explanation,  the  remainder  of  this  section  will  focus  on  this  di¬ 
mension,  providing  examples  of  weapon  systems  when  possible.*® 

Figure  2. 1  summarizes  the  conceptual  framework  developed  thus  far, 
and  places  the  taxonomy  of  goals  within  it.  Prototyping  addresses 
various  types  of  risk  and  uncertainty  by  generating  information  that 
improves  the  management  of  that  uncertainty.  The  taxonomy  pre¬ 
sented  in  Figure  2.1  is  a  hierarchy.  The  first  level  concerns  the 
overall  purpose  of  prototyping  in  the  program;  the  second,  the  specific 
objectives  of  particular  prototypes.  In  decreasing  order  of  detail, 
these  two  levels  relate  to  the  kind  of  information  that  prototyping 
generates,  and  together  constitute  what  we  refer  to  as  the  goals  of  a 
prototyping  strategy. 


*®As  described  in  Section  3,  the  progr  im  data  come  from  available  public  literature 
and  a  survey  of  program  managers  and  are  deficient  in  many  respects.  In  no  case  was 
there  enough  information  to  categorize  a  prototype  with  perfect  confidence. 
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Figure  2.1 — A  Prototyping  Taxonomy 


Level  One:  Overall  Purpose 

The  first  level  categories,  denoted  overall  purpose,  represent  the  gen¬ 
eral  purposes  of  the  prototyping  phase  and  are  the  most  aggregate 
classification  level.  Overall  purposes  are  closely  related  to  the  ex¬ 
pected  benefits  of  prototyping  and  the  decision  stage  of  the  program. 

•  Technology  Viability:  Generating  basic  technical  information  to 
reduce  technological  risk  in  a  general  sense.  These  are  “building 
block”  prototypes,  intended  to  add  to  the  general  knowledge  base. 
They  generally  occur  very  early  in  a  program,  often  before  demon¬ 
stration/validation  at  Milestone  I,  or  even  outside  the  normal 
weapon  system  program  structure.  No  military  mission  needs  to 
be  specified. 

•  Technology  Demonstration:  Exploring  the  possible  performance 
envelope  of  a  system.  Prototypes  in  this  category  are  often  used  to 
explore  the  usefulness  of  a  new  design  or  concept  in  performing  a 
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specific  mission,  or  to  demonstrate  a  particular  application  of  tech¬ 
nology.  They  may  also  be  used  to  generate  or  preserve  design  and 
concept  options  as  a  hedge  against  threat  uncertainty.  Missions  or 
functional  requirements  are  specified.  These  prototypes  may  occur 
early  in  the  program  in  concept  exploration  or  validation  phases  at 
a  time  when  design  is  not  frozen.  Production  of  an  operational  sys¬ 
tem  is  often  anticipated. 

•  System  Design  I  Performance  Validation:  Involving  design  and 
performance  specifications  or  requirements.  Also  included  here  are 
demonstrations  of  the  ability  to  meet  a  specified  threat,  contract 
specifications,  and  producibility  concerns.  Missions  are  specified, 
often  in  detail,  and  there  is  an  expectation  of  production.  Opera¬ 
tion,  support,  and  logistics  are  also  of  concern.  This  category  might 
also  be  called  “engineering,”  since  these  prototypes  are  often  fabri¬ 
cated  as  part  of  advanced  development  or  full-scale  development  ef¬ 
forts. 

•  Marketing:  Having  to  do  directly  with  selling  a  product  or  support¬ 
ing  a  proposal.  These  prototypes  are  often  close  to  production  con¬ 
figuration  or  are  missionized.  Competition  is  a  frequent  theme 
here.  These  can  be  part  of  any  decision  phase  prior  to  production; 
they  can  also  exist  outside  the  program  milestone  structure.  There 
is  a  definite  expectation  (or  hope)  of  production.  Missions  do  not 
need  to  be  specified,  though  the  prototype  is  oriented  toward  a 
specific  functional  requirement.  These  prototypes  are  sometimes 
funded  by  private  industry. 

Normally,  only  one  main  purpose  is  relevant  to  a  single  program. 
Which  purpose  that  is  depends  on  the  ranking  of  the  key  uncertain¬ 
ties.  Protot)q)es  in  the  technology  viability  category  are  most  often 
associated  with  basic  technical  uncertainties.  Prototypes  in  the  tech¬ 
nology  demonstration  category  are  associated  with  both  technical  and 
target  uncertainties.  Almost  any  kind  of  prototype  can  be  associated 
with  internal  program  uncertainties.  System  design  prototypes  are 
more  often  associated  with  some  forms  of  target  and  internal  uncer¬ 
tainties.  Marketing  prototypes  can  address  all  four  types  of  uncer¬ 
tainties. 

Given  the  overlap  between  the  various  categories  of  risk  and  main 
purposes,  it  seems  useful  to  allow  for  secondary  purposes.  Using  the 
same  set  of  categories  as  main  purposes,  the  secondary  purpose  des¬ 
ignation  is  intended  to  capture  those  aggregate  level  goals  that  may 
be  less  important  than  the  primary  purpose,  but  still  represent  an 
important  focus  of  the  prototype  program. 


15 


Level  Two:  Specific  Objectives 

The  specific-objectives  level  explicitly  defines  many  possible  uses  for 
prototypes  and  recognizes  that  one  prototype  may  serve  several  objec¬ 
tives,  which  can  be  ranked  by  importance.  Additionally,  each  proto¬ 
type  in  a  development  program  may  be  associated  with  a  different  set 
of  specific  objectives.  Specific  objectives  relate  to  the  rationale  under¬ 
lying  fabrication  of  the  prototype  and  to  the  specific  information  gen¬ 
erated.  Eleven  specific  objectives  are  listed: 

•  Experimental:  Demonstrates  a  new  idea,  a  new  technology,  or  an 
existing  technology  in  a  new  application.  This  usually  occurs  very 
early  in  the  program  and  may  not  have  particular  mission  or  pro¬ 
duction  expectations. 

•  Exploratory:  Evaluates  the  possible  performance  envelope  or  tests 
the  feasibility  of  a  performance  requirement.  May  not  have  a  mis¬ 
sion  specified  or  expectations  of  production,  but  does  have  explicit 
performance  goals.  This  usually  occurs  in  the  concept  or  validation 
phase. 

•  Feasibility:  Demonstrates  performance  objectives  in  reference  to  a 
specific  mission.  This  usually  occurs  in  the  validation  phase, 
though  production  may  not  be  expected. 

•  Competitive:  Used  to  improve  source  selection  decisions  in  valida¬ 
tion  or  FSD  phases.  Production  is  anticipated. 

•  Developmental:  Determines  tactical  or  operational  suitability  and 
utility  for  military  uses.  May  occur  in  the  concept  or  validation 
phases.  This  is  the  missionized  version  of  an  experimental  proto¬ 
type. 

•  Political:  Achieves  some  political  or  corporate  strategy  objective, 
demonstrates  attainment  of  a  political  objective,  or  responds  to  a 
politically  established  requirement.  This  can  occur  throughout  the 
decision  process,  though  it  occurs  most  often  in  validation  or  FSD. 

•  Integration:  Tests  subsystem  matching  and  full  system  operation. 
May  be  part  of  the  concept,  validation,  or  FSD  phases.  Specific 
mission  or  functional  requirement  exists. 

•  Preproduction:  Tests  production  configuration  after  design  freeze, 
usually  during  FSD.  Producibility  concerns  are  relevant.  Full-rate 
production  is  expected. 

•  Missionized:  Evaluates  performance  with  respect  to  specified 
threat  using  fully  integrated  system.  This  may  occur  in  concept, 
validation,  or  FSD  phases. 
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•  Operational:  Tests  operational  suitability  of  fully  integrated  sys¬ 
tem,  including  reliability,  availability,  and  maintainability  charac¬ 
teristics.  Also  used  for  doctrine  development  and  integrated  logis¬ 
tics  support  planning. 

•  Upgrade:  Tests  or  demonstrates  subsystem  improvement  to  exist¬ 
ing  system  in  operational  use.  Occurs  either  during  the  production 
phase  of  existing  platforms  or  as  a  separate  retrofit  program.  The 
upgrade  itself  may  be  part  of  a  concept,  validation,  or  FSD  phase. 

Notice  that  as  we  move  down  through  the  list  of  specific  objectives, 
the  prototype  increasingly  resembles  final  production  configuration, 
occurs  later  in  the  program,  is  more  fully  integrated  (i.e.,  includes  a 
complete  set  of  subsystems),  and  is  less  austere.  Also  note  that  two  of 
the  objectives,  competitive  and  political,  differ  from  the  others  fun¬ 
damentally.  These  two  relate  more  to  the  acquisition  process  and 
programmatic  considerations  than  to  technology. 

It  should  also  be  noted  that  several  previously  mentioned  potential 
benefits  of  prototyping  are  not  explicitly  included:  improving  cost  and 
schedule  estimates,  allowing  more  informed  trade-off  decisions,  and 
providing  a  hedge  against  threat  uncertainty.  Though  we  consider 
these  benefits  of  prototypes  as  important  as  the  rest,  and  a  properly 
executed  prototype  program  may  actually  satisfy  those  goals,  in  the 
current  acquisition  environment  virtually  no  prototype  program  has 
these  as  a  main  purpose. 

UNDERSTANDING  THE  FRAMEWORK 

To  understand  how  this  taxonomy  would  be  applied,  consider  several 
examples.  The  first  National  Aerospace  Plane  {X-30),  which  has  as 
its  goal  to  demonstrate  the  maturity  of  hypersonic  flight  technology, 
would  be  considered  a  technology  viability  prototype,  with  specific  ob¬ 
jectives  relating  to  the  experimental  and  feasibility  categories.  The 
tilt-rotor  XV- 15,  a  vertical  takeoff  and  landing  air  vehicle,  would  be 
classified  as  a  technology  demonstrator  prototype  whose  specific  ob¬ 
jectives  were  exploratory  and  developmental.  The  UH-60  Black 
Hawk  helicopter  would  be  classified  as  a  system  design/performance 
validation  prototype;  specific  objectives  included  preproduction  and 
operational  goals,  as  well  as  competition.  Northrop’s  F-20,  a  rela¬ 
tively  low-cost  multirole  fighter  aircraft,  was  a  marketing  prototype. 
Its  specific  objectives  fell  into  the  preproduction,  competitive,  and  op¬ 
erational  categories. 
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Under  this  framework,  there  are  many  different  types  of  prototyping 
strategies,  depending  on  the  kind  of  risk  involved  and  the  decision  to 
be  affected.  As  noted  above,  each  system  has  only  one  main  purpose 
but  may  have  several  specific  objectives.  The  idea  of  this  taxonomy  is 
to  capture  the  basic  rationale  for  building  the  prototype  with  the 
main  purpose  categories,  then  to  define  as  many  of  the  specific  objec¬ 
tives  as  possible. 

There  are  strong  relationships  between  the  elements  of  the  frame¬ 
work  with  respect  to  the  various  dimensions  and  characteristics  that 
define  prototyping.  For  example.  Table  2.2  indicates  that  certain 
specific  objectives  are  intuitively  associated  with  particular  main 
purposes.  That  relationship  results  from  the  kinds  of  risk  and  uncer¬ 
tainty  addressed  in  each  purpose  and  objectives  category,  as  well  as 
the  level  of  system  integration  and  phase  of  the  program.  Similarly, 
some  combinations  of  main  and  secondary  purposes  are  expected  to  be 
more  common  than  others.  For  instance,  technology  demonstration 
as  a  main  purpose  seems  consistent  with  system  design  as  a  sec- 
ondaiy  purpose,  but  the  combination  of  technology  viability  and  mar¬ 
keting  seems  unlikely.  There  are  also  combinations  of  objectives  that 
are  intuitively  more  likely  than  other  combinations.  Experimental, 
exploratory,  and  feasibility  are  all  related  in  that  they  concern  tech¬ 
nology  and  basic  applications.  Preproduction,  missionized,  and 
operational  prototypes  overlap  to  the  extent  that  they  reflect  the  full 
system.  However,  it  is  very  unlikely  that  experimental  and 
operational  objectives  would  be  found  in  the  same  prototype.  The 
message  here  is  that  achievement  of  one  particular  objective  may 
exclude  achievement  of  some  other  objectives.  In  other  words, 
prototypes  are  narrowly  focused  on  specific  issues  important  to  a 
particular  program,  and  thus  are  more  likely  to  produce  information 
useful  to  progfram  managers. 


Table  2J2 

Common  Purpose-Objective  Associations 


Technology  Viability 

•  experimental 

•  exploratory 

System  Design/ 
Performance  Validation 

•  integration 

•  preproduction 

•  missionized 

•  operational 

•  upgrade _ 


Technology  Demonstration 

•  feasibility 

•  developmental 

Marketing 

•  political 

•  competitive 
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As  we  have  said,  there  is  also  a  theoretical  relationship  between  main 
purpose/specific  objectives  and  the  program  phase  in  which  prototyp¬ 
ing  occurs.  Table  2.3  outlines  this  relationship  by  specific  objective. 
Although  there  are  situations  when  these  relationships  do  not  hold, 
the  pattern  is  fairly  clear.  Experimental  prototypes  would  not  be  ex¬ 
pected  to  occur  during  FSD,  and  operational  prototypes  do  not  usually 
occur  during  concept  exploration.  Notice  that  prototypes  never  occur 
during  production;  by  the  time  there  is  a  commitment  to  production, 
most  major  decisions  affecting  design,  technology,  and  operational 
utility  have  already  been  made.** 

Another  way  to  illustrate  the  relationships  between  the  various  spe¬ 
cific  objectives  is  to  compare  each  objective  in  terms  of  the  dimensions 
of  prototyping  listed  earlier.  Table  2.4  shows  that  similarity  in  proto¬ 
typing  dimensions  implies  some  overlap  in  goals.*^  Notice  in  Table 


Table  2,3 

Interaction  of  Objectives  and  Process 


Specific  Objective 

Process  Phase 

Concept 

(Milestone  0  - 1) 

DemAfal 
(Milestone  I  - 11) 

FSD 

(Milestone  II  -  IID 

Experimental 

X 

Exploratory 

X 

X 

Feasibility 

X 

Competitive 

X 

Developmental 

X 

X 

Political 

X 

Integration 

X 

Preproduction 

X 

Miss  ionized 

X 

Operational 

X 

Upgrade 

X 

**The  exception  is  when  a  production  facility  or  critical  production  tooling  is 
prototyped  in  some  form.  Upgrades  and  modiflcation  programs  would  be  classified 
with  respect  to  the  development  stage  of  the  subsystem  b^ng  incorporated. 

*^There  are  always  a  few  exceptions  to  any  rule.  For  instance,  while  it  may  be  clear 
that  the  goals  of  the  National  Aerospace  Plane  (NASP)  program  are  properly  classified 
as  technology  viability  with  experimental  and  feasibility  objectives,  it  is  also  clear  that 
NASP  is  not  austere  in  terms  of  budget  or  management.  Further,  the  NASP 
integration  level  is  probably  not  *low*  as  might  be  implied  by  the  chart;  propulsion, 
aerodynamic  shapes,  and  flight  controls  will  need  to  be  highly  integrated.  Thus  it  is 
demonstrated  that  properly  classifying  a  given  prototype  involves  consideration  of  its 
technical,  political,  and  economic  characteristics;  the  matrix  in  Table  2.4  is  meant  only 
to  be  suggestive. 


Table  2.4 
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2.4  that  objectives  such  as  experimental  and  feasibility,  or  mission- 
ized  and  operational,  are  fairly  close  in  terms  of  the  dimensions;  thus, 
those  objectives  overlap  substantially.  Also  notice  that  objectives 
such  as  exploratory  and  preproduction  are  at  opposite  extremes, 
indicating  for  instance  that  a  prototype  with  a  main  objective  of  dem¬ 
onstrating  and  exploring  the  reasonableness  of  a  performance 
requirement  will  not  have  characteristics  necessary  to  test  the  final 
production  design. 

VALUE  AND  LIMITS  OF  THE  FRAMEWORK 

The  categorization  of  prototypes  is  a  highly  subjective  exercise.  It  is 
useful  only  to  the  extent  that  we  learn  something  about  which  charac¬ 
teristics  of  prototyping  strategy  are  necessary  to  achieve  specified 
goals.  From  the  dimensions  defined  above,  we  can  derive  two  major 
lessons:  (1)  prototyping  strategies  are  focused  on  particular  issues 
identified  as  key  risks  and  uncertainties  in  the  program,  and  (2)  there 
are  differences  between  prototypes  and  associated  acquisition  strate¬ 
gies  that  suggest  that  attainment  of  certain  objectives  precludes  at¬ 
tainment  of  others. 

It  should  also  be  noted  that  none  of  the  above  discussion  is  meant 
to  imply  that  there  are  generic  strategies  that  can  be  applied  to  de¬ 
fined  circumstances.  Each  program  is  unique.  While  classification 
schemes  are  useful,  an  effective  prototyping  strategy  must  be  tailored 
to  reflect  this  real-world  variation. 


3.  STUDY  DESIGN  AND  DATA 


This  analysis  included  two  separate  data  collection  efforts — a  litera¬ 
ture  review  and  a  survey  of  U.S.  government  program  managers.  The 
resulting  databases  cover  a  large  number  of  diverse  programs  and 
variables,  and  support  a  broadly  scoped  analysis  of  prototyping  expe¬ 
rience. 

The  literature  review  included  a  broad-ranging  examination  of  proto¬ 
typing  concepts  and  issues,  as  well  as  a  collection  of  general  pro¬ 
grammatic  information  on  specific  prototype  programs.  This  sup¬ 
ported  the  development  of  the  conceptual  framework  described  in 
Section  2  and  the  construction  of  a  large  database  of  programmatic 
information  on  287  programs  spanning  the  period  from  1960  to  1988. 
Sources  for  this  data  are  given  in  the  bibliography,  and  the  list  of 
programs  included  in  the  database  are  provided  in  Table  A.1. 

The  second  data  collection  effort  involved  a  formal  questionnaire  sent 
to  government  program  managers.  Responses  from  41  programs  al¬ 
lowed  construction  of  a  database  of  programmatic  characteristics  for 
those  systems.  This  data  includes  both  quantitative  and  qualitative 
information.  To  our  knowledge,  such  a  database  has  never  before 
been  developed.  Table  A.2  lists  the  programs  that  responded  to  the 
survey.  The  survey  itself  is  reproduced  in  Appendix  C. 

METHODOLOGY 

The  conceptual  framework  presented  in  Section  2  provided  guidance 
and  consistency  for  the  two  data  collection  efforts.  In  both  cases,  in¬ 
formation  was  collected  that  enabled  exploration  of  relationships  be¬ 
tween  programmatic  characteristics  that  constitute  prototyping 
strategies  and  allowed  tracking  of  broad  trends  in  those  relationships. 
To  a  considerable  extent,  the  data  collected  from  both  the  literature 
review  and  the  program  manager  survey  were  intended  as  a  rough 
test  of  the  reasonableness  of  the  conceptual  framework. 

By  design,  the  two  databases  are  complementary,  each  having  com¬ 
parative  advantages.  The  literature  review  produced  programmatic 
data  that,  while  insightful,  were  unsatisfying  in  terms  of  providing 
details  of  prototyping  strategies.  The  survey  was  intended  to  obtain 
that  additional  detail,  as  well  as  gather  supplemental  data  on  the 
characteristics  of  prototyping  programs  and  improve  the  quality  of  in¬ 
formation  available  to  analyze  past  prototyping  experience.  The  sur- 
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vey  asked  for  the  same  information  that  was  generated  as  part  of  the 
literature  review.  Thus  for  certain  sets  of  variables,  the  data  and 
analysis  for  the  literature  review  and  the  survey  are  directly  compa¬ 
rable. 

Many  different  journals  and  reports  were  examined  in  an  attempt  to 
compile  a  list  of  programs  incorporating  prototyping  activities  since 
1960.  Additionally,  information  relating  to  the  characteristics  of  the 
prototype  was  collected.  This  information  was  used  to  categorize  the 
prototype  program  within  the  framework  developed  previously.  Gen¬ 
eral  program  information  was  also  of  interest,  including  the  program 
phase  in  which  the  prototype  occurred,  the  management  agency  or 
agencies  involved,  and  basic  cost  and  schedule  information. 

The  survey  was  composed  of  a  worksheet  questionnaire  sent  to  85 
U.S.  government  program  managers  of  current  systems  where  there 
was  some  indication  that  prototyping  was  or  will  be  used.  We  re¬ 
ceived  43  responses  for  41  programs.* 

The  literature  review  can  cover  a  very  large  number  of  programs, 
though  in  scant  detail.  The  program  manager  survey  has  the  advan¬ 
tage  of  collecting  primary  information,  assessments,  and  opinions  on 
current  prototyping  progprams  from  the  government  participants. 
This  mitigates  any  bias  that  might  result  from  our  interpretation  of 
the  literature;  the  bias  in  the  survey  reflects  that  of  the  respondents. 
The  questionnaire  covered  more  aspects  of  the  programs  in  more  de¬ 
tail,  and  included  both  quantitative  and  qualitative  responses.  The 
qualitative  responses  were  particularly  interesting  as  they  reflected 
managers’  experiences. 

DATABASE  DESCRIPTION 

Table  3.1  indicates  the  range  of  information  that  was  collected  by 
both  the  literature  review  and  the  program  manager  questionnaire. 
'There  are  three  types  of  variables:  quantitative  (cost,  schedule,  etc.), 
categorical,  and  indicator.  The  questionnaire  collected  the  same  data 
gathered  through  the  literature,  plus  considerably  more  on  a  wider 
range  of  programmatic  characteristics.  In  all  these  cases,  the  mea¬ 
sures  used  in  both  the  literature  review  and  the  survey  were  identi¬ 
cal. 


*The  Navstar  Global  Positioning  System  (GPS)  program  office  provided  three 
responses,  one  each  for  satellite,  user  equipment,  and  ground  control  segments  of  the 
program.  Each  had  its  own  acquisition  strategy,  including  contracting  and 
development  tasks. 
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V 

Variables  Used  in  Analysis 

Included  in 

Liter- 

PM  Sur- 

Variable 

Description 

ature 

vey 

Program  name 

System  designation 

X 

X 

Position 

Position  of  respondent 

X 

Experience 

No.  of  years  at  position 

X 

Weapon  type 

System  type  classification 

X 

X 

Management  agency 

Organization  responsible  for  funding  and 

management 

X 

X 

Meaning 

Meaning  of  term  prototype 

X 

Distinguish 

Difference  between  prototype  and  PSD  article 

X 

Definition  (1) 

Survey-based  definition 

X 

Definition  (2) 

Derived,  narrow  definition 

X 

X 

Benefits 

Expected  benefits  of  prototyping 

X 

Effects 

Expected  effect  on  outcomes 

X 

Levels  of  prototypes 

No.  of  prototypes  by  integration  level 

x» 

X 

Choice 

Prototype  level  referenced 

X 

Program  phase 

Phase  prototyping  occurred 

X 

X 

Main  purpose 

Main  purpose  of  activities 

X 

X 

Secondary  purpose 

Second-order  purpose  of  activity 

X 

X 

Specific  objective  1 

First-order  objective  where  most  of  effort  was 

expended 

X 

X 

Specific  objective  2 

Second-order  objective 

X 

X 

Specific  objective  3 

Third-order  objective 

X 

X 

Specific  objective  4 

Fourth-order  objective 

X 

Specific  objective  5 

Fifth-order  objective 

X 

Specifications 

Method  of  requirements  specification 

X 

Performance  mod 

Requirement  modification  due  to  prototype 

testing 

X 

Contract  type 

Type  of  contract  by  program  phase 

X 

Substitution 

Prototyping  as  a  substitute  for  other  phases 

X 

Documentation 

Relative  amount  of  reporting 

X 

Layering 

No.  of  decision  layers  by  program  phase 

X 

■ 

Decision  maker 

Office  responsible  for  cost,  schedule. 

performance  decisions 

X 

Teaming 

Teaming  strategy  indicator 

X 

Prototype  cost 

Cost  of  prototyping  activities 

X 

X 

Development  cost 

RDT&E^  cost,  including  prototype 

X 

X 

Procurement  cost 

Cost  of  total  production 

X 

X 

Total  cost 

Total  program  cost 

X 

X 

Quantities 

Prototype,  FSD,  and  Production 

X' 

X 

Program  start 

Year  of  program  initiation 

X 

X 

First  operation 

Year  of  first  prototype  operation 

X 

X 

Design  time 

Months  from  program  start  to  first  prototype 

operation 

X 

X 

Prototype  phase 

Months  from  program  start  to  completion  of 

test  program 

X 

X 
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Table  3.1  (continued) 


Variable 

Description 

Included  in 

Liter-  PM 
ature  Survey 

Program  length 

Months  from  program  start  to  first  delivery 
of  production  unit 

X 

x<* 

Responsible  oHioe 

Person  who  made  decision  to  prototype 

X 

Timing 

Whether  decision  was  made  before  or  alter 

X 

Technological 

program  start 

Evolutionary  or  revolutionary 

X 

advance 

Challenge 

Major  technical  challenge 

X 

Role  of  prototype 

Role  prototype  played  in  meeting  challenge 

X 

CAD/CAM 

Whether  CAD/CAM  was  used 

X 

Tooling  difference 

Difference  between  PSD  unit  and 

X 

Skill  level 

prototype  fabrication 

Relative  skill  level  of  labor 

X 

^Literature-based  data  contain  a  single  categorical  variable;  survey  has  one  each 
for  full  system,  partial  system,  subsystem,  component,  other. 

^Research,  develop,  test,  and  evaluate. 

^Literature  data  have  prototype  quantities  only. 

*^Survey  has  calendar  dates  for  the  following’.  Milestones  1,  II,  Ilia,  Illb,  design 
start,  fabrication  start,  first  operation,  test  objectives  achieved.  Various  intervals  can 
be  calculated. 

This  research  postulated  a  few  basic  relationships  and  trends  with  re¬ 
spect  to  prototyping.  We  felt  that  prototyping  strategies  could  be  dis¬ 
tinguished  from  other  acquisition  strategies;  that  they  could  be  char¬ 
acterized  by  a  few  basic  programmatic  variables  (e.g.,  goals,  timing, 
and  level  of  integration);  that  these  basic  elements  are  related  to 
more  detailed  programmatic  characteristics;  and  that  prototyping 
strategies  have  changed  over  time  as  a  result  of  a  changing  acquisi¬ 
tion  environment.  Additionally,  prototyping  was  expected  to  be  asso¬ 
ciated  with  relatively  better  program  outcomes  in  terms  of  cost 
growth  and  schedule  slip. 

The  type  of  information  collected  in  both  the  literature  review  and 
program  manager  survey  relates  to  those  postulates.  Basic  pro¬ 
grammatic  data  on  cost,  schedule,  decision  process,  and  contracting 
strategies  allow  a  basis  for  understanding  prototyping  as  an  acquisi¬ 
tion  strategy.  Categorical  variables  relating  to  the  definition  of  proto¬ 
typing,  expected  benefits,  and  goals  allowed  characterization  of  spe¬ 
cific  prototyping  strategies.  The  time-based  information  supports  the 
objective  of  understanding  the  extent  to  which  prototyping  strategies 
have  changed  as  the  acquisition  environment  has  changed.  Data  on 
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program  cost  and  schedule  outcomes  support  a  first-cut  analysis  of 

the  effect  of  prototyping  on  those  outcomes.  Some  of  the  variables 

listed  in  Table  3.1  are  briefly  described  below: 

•  Program  name  is  a  tracking  variable  that  allows  identification  of 
any  outliers. 

•  The  position  and  experience  variables  are  used  as  data  quality  con¬ 
trol  checks  for  the  program  manager  survey.  For  instance,  the  re¬ 
sponses  of  a  program  manager  who  has  been  on  the  job  for  three  or 
more  years  should  qualitatively  carry  more  weight  than  those  of  a 
manager  who  has  less  than  one  year  of  experience. 

•  The  categories  of  weapon  type  used  in  the  literature  review  data¬ 
base  include  fixed-wing  aircraft,  helicopters,  land  vehicles,  mis¬ 
siles,  subsystems,  and  other.  The  subsystem  category  includes 
guns,  avionics,  engines,  etc.  The  other  category  includes  space  sys¬ 
tems  and  ships.  The  smaller  number  of  programs  in  the  survey 
could  reasonably  support  fewer  categories:  aircraft,  helicopters, 
vehicles,  electronic  systems,  and  other. 

•  The  management  agency  is  simply  the  organization  responsible  for 
the  program.  Categories  used  in  the  literature  survey  include  the 
Army,  Navy  and  Marine  Corp,  Air  Force,  Defense  Advanced  Re¬ 
search  Projects  Agency  (DARPA),  joint  U.S.,  other  U.S.,  joint 
U.S./foreign,  foreign  government,  foreign  private,  and  U.S.  private. 
The  questionnaire  uses  the  three  U.S.  services  and  a  joint  program 
category. 

•  The  number  of  prototypes  is  the  number  of  prototype  articles  fabri¬ 
cated.  For  competitive  programs,  this  includes  all  contractors.  The 
survey  collected  this  information  by  level  of  integration  as  well  as 
for  the  specific  level  of  prototyping  chosen  for  discussion  by  the 
survey  respondent. 

•  The  level  of  integration  refers  to  the  portion  of  the  total  system  that 
was  prototyped.  There  are  three  categories  here:  full  system, 
which  includes  all  key  subsystems;  partial  system,  which  includes 
only  one  or  two  subsystems  integrated  into  a  platform  (e.g.,  an  en¬ 
gine-airframe  combination);  and  subsystem,  which  includes  sub¬ 
systems  prototyped  independent  of  a  platform. 

•  The  main  and  secondary  purposes  and  the  three  specific  objectives 
are  defined  according  to  the  taxonomy  presented  earlier. 

•  The  phase  identifies  that  part  of  the  program  (based  on  decision 
milestones)  that  incorporated  a  prototype. 
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•  Two  calendar  dates  are  included:  the  program  start  date,  deter¬ 
mined  as  either  the  first  decision  milestone  or  when  a  technical  de¬ 
velopment  became  focused  enough  to  form  a  program  office,  and 
the  date  of  first  prototype  operation. 

•  The  schedule  data  show  months  from  program  start  and  were 
based  on  decision  milestones  or  contract  award  dates. 

•  The  cost  data  are  in  millions  of  constant  dollars  (FY82  for  the  liter¬ 
ature  data,  FY89  for  the  questionnaire)  and  are  used  as  a  proxy  for 
program  size  and  level  of  effort.  Actual  reported  costs  or  the  most 
current  estimates  were  used. 

•  The  definitional  indicator  allows  an  important  problem  to  be  ad¬ 
dressed:  the  variability  in  definitions  used  in  the  literature.  Based 
on  the  more  narrow  definition  of  a  prototype  presented  in  the  pre¬ 
vious  section,^  programs  were  coded  as  either  meeting  this  criteria, 
not  meeting  it,  or  providing  insufficient  information  to  support  a 
finding. 

Some  data  obtained  in  the  survey  were  not  available  in  the  literature. 
Most  of  these  concern  either  aspects  of  the  programs  potentially  re¬ 
lated  to  prototyping  strategies  (e.g.,  contract  type,  decision  layers, 
amount  of  documentation)  or  asked  the  respondent  to  qualitatively 
describe  the  rationale  behind  prototyping  in  the  program.  For  in¬ 
stance,  questions  regarding  expected  benefits,  effects  on  program  out¬ 
comes,  and  the  difference  between  prototypes  and  other  test  articles 
resulted  in  responses  that  provide  additional  insight  into  prototyping 
activities.  Those  responses  are  reflected  in  the  discussion  in  the  fol¬ 
lowing  sections. 

The  data  were  coded  in  a  way  suitable  for  some  basic  statistical 
analysis,  mostly  frequency  distributions  and  cross-tabulations.  Be¬ 
cause  of  the  nature  of  the  data  collected,  and  the  small  number  of 
observations  for  some  subsets  of  the  database,  we  relied  more  on 
simple  correlations  and  evaluation  of  tables  and  graphs  rather  than 
formal  statistical  tests  for  significance,  though  we  did  perform  those 
when  appropriate. 


^The  general  deflnition  of  a  prototype  used  in  this  report  is  that  it  is  an  article 
fabricated  in  the  expectation  of  change.  This  implies  that  the  results  of  prototyping 
activities  contribute  to  technical  or  programmatic  decisions  prior  to  full-rate 
production.  A  prototype  has  objectives  other  than  only  demonstrating  satisfaction  of 
contract  specifications. 
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LIMITS  OF  THE  STUDY 

The  data  used  here  reflect  several  problems  and  limitations.  First, 
the  inconsistent  use  of  the  term  prototype  in  the  literature  affects  the 
quality  of  that  data.  Because  of  the  very  large  variance  in  definitions, 
corrections  could  not  be  made  without  degrading  the  data.  Therefore, 
programs  were  included  in  the  database  if  one  or  more  sources  indi¬ 
cated  that  the  program  incorporated  prototyping  activities  of  some 
kind.  This  problem  is  partially  resolved  by  the  definitional  variable 
described  alMve. 

There  are  also  considerable  gaps  in  the  data.  Very  few  programs 
have  complete  information  of  relatively  good  quality  for  all  variables. 
This  means  that  for  some  of  the  analysis,  particularly  that  focusing 
on  trends  in  the  cost  and  schedule  variables,  there  are  few  data 
points.  To  mitigate  the  small  database  problem,  much  of  the  analysis 
presented  in  Sections  4  and  5  uses  the  data  generated  through  the 
literature  review.  This  larger  database  is  more  amenable  to  statisti¬ 
cal  and  graphical  analysis.  Nonetheless,  the  same  analysis  was  per¬ 
formed  on  the  survey  data.  Some  of  the  more  interesting  or  unique 
results  are  presented  alongside  the  results  of  the  literature  review. 
Additional  survey  results  are  documented  in  detail  in  Appendix  B. 

Similarly,  the  list  of  post-1960  programs  that  constitutes  the 
database  used  in  the  analysis  is  not  complete,  but  it  is  representative 
of  prototyping  activities  over  the  time  period  of  interest.  There  are 
indications  in  the  literature  that  there  have  been  more  prototype  pro¬ 
grams  since  1960  than  those  listed  in  Table  A.l.  Additionally,  since 
we  collected  information  only  on  programs  believed  to  include  proto¬ 
typing  in  some  form,  and  examined  only  those  that  met  a  slightly 
more  specific  definition,  we  cannot  know  the  relative  frequency  of  pro¬ 
totyping  versus  nonprototyping  programs. 

Both  the  coding  and  the  taxonomy  are  subjective.  While  this  does  not 
hinder  the  analysis,  it  does  suggest  that  one  must  be  careful  when 
generalizing  outside  of  the  database  and  the  assumptions  used  in  the 
analysis.  A  few  problems  related  to  the  general  subjectivity  of  the 
taxonomy  are  worth  mentioning.  When  coding  the  variable  for  inte¬ 
gration  level,  a  judgment  had  to  be  made  regarding  the  representa¬ 
tiveness  of  the  prototype  relative  to  the  final  objective  system.  This 
was  complicated  by  the  fact  that  any  given  program  can  include  pro¬ 
totyping  during  more  than  one  phase.  In  particular,  programs  that 
included  a  series  of  prototypes,  each  increasingly  closer  to  production 
configuration,  were  difficult  to  categorize.  These  programs  were  cate¬ 
gorized  as  partial  system  prototypes,  reflecting  an  “average.”  This 
type  of  subjective  coding  problem  was  mitigated  somewhat  in  the  sur- 


28 


vey,  which  asked  the  respondent  to  specify  which  prototype  (if  there 
was  more  than  one)  was  the  object  of  the  responses.  Additionally,  the 
schedule-related  variables  are  subject  to  some  uncertainty  and  incon¬ 
sistency  across  programs.  Identifying  functionally  equivalent  mile¬ 
stones  has  proved  to  be  a  difficult  task. 

As  mentioned  earlier,  the  choice  of  research  approach  constrains  both 
the  kind  of  analysis  that  can  be  performed  and  the  conclusions  that 
can  be  drawn  from  it.  The  large-sample  empirical  analysis  adopted 
here  allows  us  to  address  questions  concerning  the  components  of  a 
prototyping  strategy,  how  those  components  relate  to  each  other,  and 
how  prototyping  strategies  have  been  used  in  the  past.  We  can  also 
gain  some  insight  into  the  factors  affecting  the  choice  of  strategy,  at 
least  at  the  macro-level.  We  cannot  address  questions  relating  to  the 
appropriateness  of  a  specific  prototyping  application,  or  its  effect  on 
the  program.  Those  questions  require  detailed  case  studies. 

In  spite  of  these  limitations,  an  analysis  of  this  type  seems  useful.  To 
the  best  of  our  knowledge,  this  is  the  first  time  that  such  a  large 
database  of  prototyping  activities  has  been  compiled,  and  it  does  yield 
some  interesting  insights  into  the  nature  of  prototyping  and  its  role  in 
weapon  system  acquisition. 

DESCraPTION  OF  SAMPLE 

The  literature  review  yielded  data  on  287  programs  spanning  the 
time  period  from  1960  to  early  1988.  Programs  were  included  if  one 
or  more  sources  in  the  literature  indicated  that  the  program  incorpo¬ 
rated,  or  will  incorporate,  prototyping  activities,  and  if  the  first  oper¬ 
ation  of  the  prototype  article  occurred  in  1960  or  later. 

The  prototyping  worksheet  questionnaire  was  sent  to  85  U.S.  gov¬ 
ernment  program  managers:  43  responses  were  received,  covering  41 
programs.  These  programs  were  in  various  stages  of  development  or 
procurement  at  the  time  the  survey  was  conducted  (May  to  October 
1988).  Of  those  41  systems,  32  overlap  the  larger  literature  review 
database,  though  in  some  cases  the  prototype  referred  to  in  the  sur¬ 
vey  is  not  the  same  as  the  one  identified  in  the  literature.^ 
Nonetheless,  the  data  are  compatible  in  that  the  metrics  used  are 
similar. 


®The  nine  programs  not  overlapping  are:  Advanced  Attack  Helicopter — ^Aircraft 
Survivability  Equipment  (AAH  ASE),  Mobil  Subscriber  Equipment  (MSE),  Forward 
Anti-Aircraft  Defense  Command  and  Control  (FAAD  C^I),  9mm  Pistol,  Pershing  11, 
Advanced  Attack  Helicopter — Automated  Test  Equipment  (AAH  ATE),  Mk  XV  IFF, 
Short-Range  Attack  Missile  (SRAM)  K,  IR  Maverick. 
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As  mentioned  earlier,  the  criteria  for  including  programs  in  the 
database  present  some  definitional  problems,  as  the  term  prototype  is 
not  used  consistently  in  the  literature.  To  reduce  the  influence  of 
definitional  inconsistencies,  a  variable  was  developed  that  allows  the 
data  to  be  sorted  based  on  whether  the  program  meets  the  more  nar¬ 
row  definition  of  prototype  begun  in  the  previous  section: 

A  protot3rpe  is  a  product  (hardware  and/or  software)  that  allows  hands- 
on  testing  in  a  realistic  environment.  In  scope  and  scale,  it  represents  a 
concept,  subsystem,  or  production  article  with  potential  utility.  It  is 
built  in  the  expectation  of  change,  and  is  oriented  toward  generating  in¬ 
formation  improving  technical  and  programmatic  decision  making.  It 
has  purposes  and  specific  objectives  other  than  simply  demonstrating 
that  the  article  meets  development  contract  specifications.  The  results 
of  prototype  testing  are  used  in  subsequent  decisions,  prior  to  the  pro¬ 
duction  decision,  influencing  system  design  and  requirements  formula¬ 
tion,  operational  utility,  and  cost  and  schedule  estimates. 

The  definition  was  applied  consistently  within  the  limits  of  available 
data.  For  instance,  programs  in  which  production  commitment  was 
made  at  the  same  time  as  the  development  contract  were  not  consid¬ 
ered  prototypes  unless  an  article  was  fabricated  and  tested  prior  to 
the  development  contract  award. 

Figure  3.1  presents  the  distribution  of  this  definitional  variable  for 
both  the  literature  and  survey  data.  The  dominant  result  for  the  lit¬ 
erature  data  is  that  for  most  of  the  programs,  there  was  not  enough 
information  available  to  determine  whether  the  article  referred  to  as 
a  prototype  met  the  criteria.  This  result  was  expected,  given  the  poor 
data  availability  regarding  how  the  so-called  prototype  was  used  in 
the  program.  However,  there  were  still  enough  programs  that  met 
the  criteria  to  be  significant.  In  the  remainder  of  this  analysis,  unless 
otherwise  stated,  all  results  are  for  the  total  database.  If  the  distri¬ 
butions  for  the  programs  that  meet  the  narrow  definition  differ  signif¬ 
icantly  from  the  total  database,  this  will  be  indicated  together  with 
the  direction  of  the  change.  We  assumed  that  programs  with  insuffi¬ 
cient  information  were  still  prototypes,  but  less  information  regarding 
the  role  of  the  prototype  was  available.  It  turns  out  there  is  essen¬ 
tially  no  change  in  the  basic  patterns  and  relationships  examined  in 
this  analysis  as  a  result  of  sorting  by  the  definitional  variable.  The 
assumption  appears  to  be  supported. 

Most  of  the  survey  respondents  were  prototyping  programs  as  in¬ 
tended.  Two  definitional  variables  were  constructed  from  the  survey 
data.  The  first  is  a  pure  survey  response:  Ten  programs  claimed  that 
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Figure  3.1 — Distribution  of  Prototype  Definition  Designation 

no  prototyping  occurred  during  development.  The  second  (shown  in 
Figure  3.1)  is  identical  to  that  used  in  the  literature  survey:  In  this 
case,  four  programs  provided  insufficient  information  to  make  a  judg¬ 
ment,  and  eight  were  definitely  not  prototyping  programs.^  Much  of 
the  survey  data  presented  here  represent  only  the  31  programs  that 
meet  the  strict  definition  of  prototyping.  This  is  because  most  of  the 
non  prototyping  respondents  did  not  provide  any  other  information. 
To  the  extent  that  the  results  using  these  31  programs  are  similar  to 
those  of  the  literature  review,  the  treatment  of  the  literature  review 
data  with  insufficient  information  is  valid.^ 


^he  eight  nonprototyping  programs  were  FAAD  Command  and  Control,  Advanced 
Attack  Helicopter — Automated  Test  Equipment,  F-14,  Trident  II  missile,  T45TS, 
Defense  Support  Program  (DSP),  C-17A,  and  Joint  Surveillance  Target  Attack  System 
(JSTARS).  The  four  programs  with  insufficient  information  were  IR  Maverick,  Light 
Airborne  Multipurpose  System  (LAMPS)  Mk  III,  Submarine  Advanced  Combat  System 
(SUBACS),  and  hft-50  torpedo.  If  anything,  these  four  programs  appear  to  be  non¬ 
prototyping  programs.  It  is  interesting  that  the  V-22  system  claimed  no  prototyping, 
but  the  responses  continually  made  reference  to  the  XV-15  tilt-rotor  technology 
demonstrator,  thus  meeting  the  definition  of  prototyping  used  in  this  analysis. 

^As  discussed  in  more  detail  in  Section  4,  the  survey  results  are  entirely  consistent 
with  the  results  of  the  literature  review.  This  validates  the  treatment  of  the  literature 
review  data.  In  any  case,  the  actual  numbers  presented  in  the  various  tables  are  much 
less  important  than  the  patterns  and  relationships  they  illustrate.  These  relationships 
are  consistent  across  many  different  cuts  at  the  data,  implying  that  the  results  are 
fairly  robust. 
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The  literature  data  provided  a  fairly  good  distribution  across  weapon 
types,  dominated  by  fixed-wing  aircraft,  as  Figure  3.2  shows.  (The 
large  number  of  fixed-wing  aircraft  in  large  part  reflects  the  interest 
at  RAND  in  aircraft  systems,  as  much  of  the  data  used  here  comes 
from  both  published  and  internal  RAND  studies.)  However,  the  dis¬ 
tribution  does  support  examination  of  differences  in  prototyping 
strategies  across  weapon  system  types.  Figure  3.2  also  shows  a  simi¬ 
lar  system-type  distribution  for  the  survey,  dominated  by  missiles, 
aircraft,  and  electronic  systems. 

Figure  3.3  shows  the  frequency  distribution  across  the  organizations 
responsible  for  program  management.  All  three  U.S.  services  are  well 
represented  in  both  the  literature  and  survey  databases,  allowing  us 
to  test  the  hypothesis  that  protot3rping  strategies  differ  across  man¬ 
agement  agencies.  Note  that  a  sign^ificant  proportion  of  the  systems 
in  the  literature  data  are  privately  funded,  either  by  U.S.  or  foreign 
firms.  Interestingly,  these  private  ventures  are  spread  almost 
equally  across  the  various  system  types,  indicating  a  wide  spread  of 
interests  and  willingness  to  fund  prototyping  activities.  All  of  the 
joint  programs  in  the  survey  were  Air  Force  led. 
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Figure  3.2 — Sample  Distribution  by  Weapon  System  Type 
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The  survey  was  administered  only  to  U.S.  government  program  managers, 
therefore,  there  are  no  private  industry  or  foreign  categories. 


Figure  3.3 — Sample  Distribution  by  Management  Agency 


The  distribution  across  time  of  programs  in  the  literature  database  is 
shown  in  Figure  3.4,  based  on  the  year  in  which  the  prototype  was 
first  operated.  The  median  (50th  percentile)  is  1982  for  this  data  set. 
The  database  includes  more  systems  in  more  recent  years.  There  are 
several  reasons  for  this.  First,  the  more  recent  data  are  more  readily 
available.  Second,  and  related  to  this,  interest  in  prototyping  has 
been  increasing  over  the  last  few  years.  Third,  because  prototyping 
fell  out  of  favor  in  the  1960s  and  late  1970s,  the  term  may  not  have 
been  used  even  if  developmental  activities  were  similar  to  what  we 
call  prototyping  today.  Because  of  data  gaps,  the  trend  that  this  fig¬ 
ure  implies  does  not  necessarily  reflect  prototyping  activity  trends.  In 
fact,  as  mentioned  previously,  interest  in  prototyping  tends  to  be 
cyclical.  We  are  just  now  entering  a  period  in  which  prototyping  in 
one  form  or  another  is  more  common  as  a  part  of  weapon  system  de¬ 
velopment.  It  should  be  noted  that  some  of  the  prototypes  shown  here 
had  not  yet  achieved  first  operation  at  the  time  these  data  were  col¬ 
lected.  For  instance,  neither  the  Advanced  Tactical  Fighter  (ATF)  nor 
the  Light  Helicopter  (LH)  prototypes  had  flown.® 


®Both  ATF  prototype  models  (YF-23  and  YF-23)  subsequently  flew  in  the  fall  of 
1990. 
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Year  of  first  prototype  operation 


Figure  3.4 — Distribution  of  First  Operation  Date 
(Literature  Review  Data) 


Figure  3.5  shows  the  same  distribution  over  time  for  the  survey 
database.  Again,  there  are  more  programs  in  more  recent  years 
(median  is  1986),  probably  due  to  the  nature  of  the  survey  sample: 
only  current  programs  were  surveyed,  many  of  which  began  in  the 
early  or  mid-1980s.  As  before,  some  of  the  dates  shown  here  are  es¬ 
timated  first-operation  dates. 

Table  3.2  shows  summary  statistics  for  the  more  quantitative  vari¬ 
ables  based  on  the  literature  review  data.  These  include  the  num¬ 
ber  of  prototypes  in  a  program,  and  the  time-  and  cost-based 
variables.  There  are  146  programs  in  the  database  for  which  the 
number  of  prototype  articles  could  be  determined.  The  average 
across  these  programs  is  5  articles,  with  a  maximum  of  33  in  the 
Army’s  High  Mobility  Multipurpose  Wheeled  Vehicle  (HMMWV) 
program,  which  involved  a  competition  between  three  contractors, 
each  submitting  11  prototype  articles  for  testing.  This  is  unusual  for 
a  land  vehicle  system — missile  systems  are  more  likely  to  have  more 
prototype  articles  than  other  systems. 
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Figure  3.5 — Distribution  of  First  Operation  Date 
(Program  Manager  Survey) 


Table  3.2 

Summary  Statistics  for  Assorted  Variables  (Literature  Data) 
(time  in  months  from  program  start,  costs  in  millions,  constant  1982$) 


Variable 

Unit 

No.  of 
Obs. 

Mean 

Standard 

Deviation 

Min. 

Max. 

No.  of  prototypes  in 

program 

number 

146 

5.1 

5.5 

1 

33 

Design  time 

months 

90 

29.0 

17.1 

5 

99 

Prototype  phase  time 

months 

26 

46.9 

20.9 

12 

84 

Total  program  length 

months 

34 

93.1 

34.8 

32 

177 

Prototype  cost 

!millions 

25 

190.2 

260.8 

2.1 

1076.0 

Development  cost 

!millions 

49 

1341.9 

1627.9 

21.4 

9025.1 

Procurement  cost 

!miIlions 

47 

5931.9 

7209.8 

30.0 

33543.3 

Total  program  cost 

!mi11ions 

52 

7332.1 

8988.1 

51.4 

38936.2 

Design  as  %  proto  time 

25 

60.5 

16.4 

18.7 

89.5 

Design  as  %  tot.  length 

22 

34.3 

20.5 

13.0 

46.1 

Proto  as  %  tot.  length 

15 

51.7 

18.6 

26.9 

77.9 

Proto!  as  %  RDT&E! 

9 

39.1 

31.7 

10.6 

100.0 

Proto!  as  %  proc! 

10 

15.5 

31.3 

0.6 

103.1 

P*roto!  as  %  total! 

10 

7.2 

8.2 

0.6 

26.3 

SOURCE:  Cost  and  schedule  data  are  mainly  derived  from  Selected  Acquisition  Re¬ 
ports.  These  were  supplemented  using  data  from  articles  cited  in  the  bibliography. 
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The  time-based  variables  (design  time,  prototype  phase  time,  and  to¬ 
tal  program  leng^)  are  all  calculated  in  months  from  program  start, 
defined  as  Milestone  I  or  the  equivalent.  Notice  that  design  time 
(time  from  program  start  to  first  prototype  operation)  data  were  more 
available  than  the  others,  mostly  because  first  prototype  operation 
date  is  more  easily  identified  than  completion  of  the  prototype  test 
program.  The  ranges  of  these  variables  are  considerable,  though 
their  relationship  to  each  other  is  as  expected.  Design  time  is  a  sub¬ 
set  of  prototype  phase  time,  which  in  turn  is  a  subset  of  program 
length.  Though  the  number  of  observations  decreases  when  sorting 
by  the  definitional  variable,  the  means  and  standard  deviations  are 
about  the  same. 

The  cost  variables  in  Table  3.2  indicate  high  variability  across  pro¬ 
grams.  The  prototype  cost  represents  those  costs  associated  only  with 
the  prototyping  activity.  It  is  a  subset  of  development  costs,  which 
cover  all  research,  development,  and  testing  costs  throughout  the 
program.  Procurement  cost  refers  to  the  cost  of  production.  Total 
program  costs  are  the  sum  of  development  and  procurement  costs. 
The  highest  costs  are  associated  with  aircraft.  The  F-20  is  the  most 
costly  prototype  in  the  sample;  this  privately  funded  program  in¬ 
cluded  only  costs  related  to  building  or  demonstrating  (marketing) 
the  prototype.  The  current  estimate  (as  of  1988)  of  ATF  development 
costs  represents  the  maximum  development  cost  in  the  database. 
Procurement  and  total  cost  maximums  are  also  aircraft — the  F-16 
and  the  B-IB. 

The  other  variables  listed  in  Table  3.2  are  percentage  variables  calcu¬ 
lated  from  cost  and  schedule  information  in  the  database.  They 
clearly  illustrate  a  data-availability  problem,  as  indicated  by  the  sub¬ 
stantial  decrease  in  the  number  of  observations,  particularly  for  the 
cost  data.  However,  both  the  time  and  cost  percentages  seem  reason¬ 
able.  Design  time  is  intuitively  a  large  portion  of  the  prototype  phase 
time  and  a  smaller  portion  of  total  program  length.  As  expected,  the 
cost  of  the  prototype  as  a  percentage  of  development,  procurement, 
and  total  cost  tends  to  decrease  and  is  in  the  range  of  previous  stud¬ 
ies.'^ 

Table  3.3  shows  similar  quantitative  data  based  on  the  program 
manager  survey.  These  data  are  for  the  31  prototyping  programs 


^The  exception  is  the  T-46A,  for  which  about  $100  million  was  spent  in  production 
funds,  less  than  that  spent  during  development.  The  result  is  that  the  cost  of  the 
prototype  as  a  proportion  of  procurement  cost  was  103  percent.  This  program  does  not 
meet  the  definitional  criteria  of  a  prototype  program,  and  was  canceled  before 
development  was  complete. 
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Table  3^ 

Summary  Statistics  for  Assorted  Variables  (Survey  Results) 
(prototype  programs  only;  constant  FY89$;  time  in  months) 


Variable 

Unit 

Obs. 

Mean 

Std.  Dev. 

Min. 

Max. 

No.  of  prototype  units 

number 

29 

21 

56.2 

0 

302 

Prototype  coat 

$millions 

23 

309.0 

755.5 

0 

3409.1 

RDT&E  coat 

$miIIions 

30 

1203.6 

2141.5 

0 

11363.6 

Production  cost 

$miIlions 

28 

7046.3 

11722.3 

1.3 

48782.0 

Total  program  cost 

$milIions 

22 

10079.3 

15417.9 

13.7 

52147.8 

Proto$  as  %  of  RDT&E$ 

22 

27.7 

29.5 

0 

100 

Proto$  as  %  of  prod.$ 

22 

8.6 

13.6 

0 

49.8 

Proto$  as  %  of  total$ 

22 

4.2 

5.4 

0 

24.1 

RDT&E$  as  %  of  total$ 

22 

26.3 

19.6 

0 

90.5 

Time  from  program  initiation  to: 

Milestone  I 

months 

21 

22.4 

17.9 

0 

62 

Milestone  II  (PSD) 

months 

25 

52.9 

30.8 

0 

126 

Milestone  Ilia  (LRIP) 

months 

22 

101.9 

38.2 

30 

170 

Milestone  Illb  (full  rate) 

months 

25 

106.8 

49.0 

24 

209 

Design  start 

months 

25 

42.0 

43.7 

0 

157 

Fabrication  start 

months 

24 

47.9 

42.8 

2 

159 

First  operation 

months 

24 

64.5 

46.6 

6 

163 

Objectives  achieved 

months 

25 

79.8 

53.3 

10 

210 

Total  prototype  phase 

months 

29 

43.6 

28.3 

8 

152 

Time  between: 

Milestones  I  and  II 

months 

21 

36.4 

23.1 

0 

79 

Milestones  11  and  Ilia 

months 

21 

45.8 

16.2 

21 

83 

Milestones  11  and  Illb 

months 

22 

61.9 

22.3 

20 

122 

Design  and  fabrication 

months 

23 

8.9 

7.6 

0 

28 

Fab.  and  1st  operation 

months 

23 

15.3 

9.0 

2 

32 

1st  oper.  and  obj.  ach. 

months 

24 

13.4 

11.2 

1 

47 

Concurrency  (Milestone  Ilia 

and  Obj.  Ach.) 

months 

19 

20.5 

41.0 

-40 

86 

only.  Similar  cost  patterns  emerge:  prototypes  tend  to  be  a  signifi¬ 
cant  portion  of  development  costs,  but  a  fairly  small  percentage  of  to¬ 
tal  costs.  Also  note  that  while  the  schedule  milestone  averages  (both 
decision  and  design)  tend  to  follow  the  expected  pattern,  there  is  con¬ 
siderable  variation  about  the  mean.  The  design  time  variable  used  in 
the  literature  review  is  equivalent  to  the  sum  of  the  intervals  of  time 
between  design  and  fabrication,  and  fabrication  and  first  operation; 
the  programs  in  the  survey  sample  tended  to  reach  first  prototype  op¬ 
eration  slightly  faster,  on  average.  However,  the  total  prototype 
phase  times  are  about  equal.  The  concurrency  variable,  measured 
here  as  the  difference  (in  months)  between  the  low  rate  production 
decision  and  the  time  at  which  the  test  phase  objectives  were 
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achieved  is  about  20  months,  indicating  fairly  high  concurrency  even 
in  these  prototyping  programs. 

The  cost  and  schedule  data  suggest  that,  on  average,  the  databases 
are  very  similar,  thus  enhancing  the  generalizability  of  the  results. 
Figures  3.6  and  3.7  illustrate  this.  The  average  total  program  cost  in 
both  the  literature  data  and  the  survey  are  almost  equal  at  $10  billion 
(FY89  constant  dollars).  The  average  cost  of  prototyping  activities 
varies  somewhat  more:  $190  million  in  the  literature  data  and  $309 
million  in  the  program  manager  survey.  The  differences  in  schedule 
intervals  (Figure  3.7)  are  also  relatively  small.  For  instance,  the 
length  of  the  prototyping  phase  was  46  months  in  the  literature 
database  and  44  months  in  the  survey. 

One  of  the  basic  messages  that  emerges  from  Tables  3.2  and  3.3  is 
that  there  is  considerable  program-to-program  variation  across  a  wide 
range  of  cost  and  schedule  variables,  as  indicated  by  the  rather  large 
standard  deviations  for  many  of  these  variables.  This  is  the  first  in¬ 
dication  of  a  basic  result  that  will  emerge  in  the  next  section:  Indi¬ 
vidual  programs  tend  to  have  unique  characteristics.  These  individ¬ 
ual  traits  result  in  large  variances  among  programs  along  any  single 
dimension  and  in  large  part  determine  the  role  of  prototyping  in  a 
program.  The  use  of  prototyping  can  vary  quite  widely  as  a  result. 


12,000 


Prototype  Development  Procurement  Total  program 


Cost  category 

SOURCE:  Tables  3.2  and  3.3. 


Figure  3.6 — Comparison  of  Literature  and  Survey  Cost  Data 
(average  costs  in  millions,  constant  FY  1989  dollars) 
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SOURCE:  Tables  3.2  and  3.3. 

NOTE:  For  survey,  design  time  is  time  from  design  start  to  first  prototype  operation, 
and  program  length  is  time  from  Milestone  I  to  Milestone  Ilia  or  equivalent. 
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Figure  3.7 — Comparison  of  Schedule  Intervals 
(Literature  and  Survey  Data) 


4.  ANALYSIS  OF  PROTOTYPING  STRATEGIES 


This  section  presents  the  results  of  an  analysis  of  past  prototyping 
programs  using  both  the  literature  review  and  the  program  manager 
survey  data.  There  are  five  subsections,  each  associated  with  a  par¬ 
ticular  set  of  research  questions. 

The  first  subsection  examines  the  range  of  possible  prototyping 
strategies  that  have  been  used  in  the  past,  using  the  taxonomy  devel¬ 
oped  in  Section  2.  In  particular,  the  three  basic  dimensions  of  proto¬ 
typing  strategies  are  examined:  level  of  integration,  timing  (phase  in 
which  prototyping  occurred),  and  goals  (purposes  and  specific  objec¬ 
tives).  Variations  in  the  magnitude  and  type  of  risk  in  each  program 
would  be  expected  to  generate  a  wide  variation  in  the  elements  of  a 
prototyping  strategy  oriented  at  addressing  them.  The  results  show 
that  a  wide  range  of  prototyping  strategies  has  been  used.  Prototyp¬ 
ing  is  a  complex  family  of  strategies,  not  one  simple  approach. 

We  next  examine  the  interrelationships  between  these  three  basic  el¬ 
ements  of  prototyping  strategies.  We  would  expect  to  find  some 
strong  relationships  based  on  the  role  of  the  prototype  in  the  devel¬ 
opment  process.  Certain  combinations  of  main  purposes  and  specific 
objectives,  for  instance,  might  be  associated  with  certain  levels  of 
system  integration  and  timing  within  a  program.  While  there  are 
many  possible  combinations  of  integration  level,  phase  in  which  proto¬ 
typing  occurs,  and  purposes,  a  few  combinations  appear  to  be  most 
common. 

The  next  subsection  looks  at  whether  those  common  strategies  change 
across  weapon  system  type  and/or  management  organization.  We 
might  expect  that  if  the  level  and  type  of  risk  and  uncertainty  and  the 
challenges  confronting  a  development  program  vary  across  system 
types,  then  this  would  be  reflected  by  observed  differences  in  proto¬ 
typing  strategies.  Similarly,  we  might  expect  that  each  management 
agency  would  have  a  distinct  style  that  would  be  reflected  in  the 
choice  of  prototyping  strategies  used.  In  fact,  however,  there  appear 
to  be  few  identifiable  differences  across  system  types  or  management 
agency  in  the  basic  elements  of  prototyping  strategy. 

A  host  of  programmatic  characteristics  might  be  expected  to  flow  from 
the  choice  of  a  particular  prototyping  strategy.  Therefore,  relying 
mostly  on  the  program  manager  survey,  we  have  also  examined  such 
items  as  contract  type,  the  number  of  organizational  layers  between 
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the  program  manager  and  the  relevant  higher-decision  authority,  and 
method  of  requirements  specification  to  see  if  there  are  in  fact  strong 
relationships.  As  expected,  these  aspects  of  development  programs 
tend  to  be  tailored  to  the  unique  aspects  of  the  program  risk  and  envi¬ 
ronment  in  much  the  same  way  as  the  basic  strategy  elements. 
Though  they  are  subject  to  great  variation,  programmatic  characteris¬ 
tics  appear  to  be  loosely  related  to  the  basic  elements  of  prototyping 
strategies. 

Prototyping  strategies  might  be  expected  to  change  as  a  result  of 
changes  in  the  acquisition  environment.  We  therefore  tried  to  iden¬ 
tify  changes  in  prototyping  strategy  with  respect  to  acquisition  phase, 
level  of  integration,  goals,  and  other  program  and  prototyping  charac¬ 
teristics.  For  instance,  we  expected  an  increase  in  the  relative  fre¬ 
quency  of  subsystem  prototyping  as  a  response  to  increasing  costs  of 
new  system  developments  and  increased  interest  in  upgrading  exist¬ 
ing  systems.  An  interesting  result  of  this  analysis  is  that  while  some 
minor  changes  are  identifiable,  there  have  been  no  radical  changes. 
The  basic  elements  of  prototyping  strategies  have  been  fairly  constant 
over  time.  To  the  extent  that  past  strategies  were  appropriate  to  the 
past  acquisition  environment,  similar  strategies  might  be  appropriate 
today. 

In  all  of  the  tables  and  graphs  in  this  section,  the  absolute  values  of 
the  data  are  much  less  important  than  the  patterns  and  relationships 
they  imply.  As  the  data  illustrate,  the  basic  patterns  and  relation¬ 
ships  are  consistent  across  both  the  literature  and  survey  data,  and 
across  many  different  aggregations  of  the  databases.  The  implication 
is  that  the  basic  results  presented  in  this  section  relating  to  prototyp¬ 
ing  strategies  and  time  trends  are  fairly  robust. 

To  facilitate  discussion  of  our  results,  we  have  made  a  conscious  effort 
to  limit  the  amount  of  data  presented  in  this  section.  Additional  de¬ 
tail  supporting  the  results  can  be  found  in  Appendix  B 

RANGE  OF  PROTOTYPING  STRATEGIES 

A  prototyping  strategy  relates  the  uses  and  values  of  prototypes  to 
achievement  of  program  goals.  We  have  argued  that  the  value  of  a 
prototype  in  weapons  development  is  as  a  risk  management  or  risk 
reduction  tool.  The  various  kinds  of  risk  addressed  by  the  prototjrpe 
in  large  part  define  the  purposes  and  objectives  (e.g.,  goals)  of  the 
prototyping  activities.  These  risks  also  define  the  other  characteris¬ 
tics  of  a  prototyping  strategy:  The  program  phase  in  which  prototyp¬ 
ing  occurs,  and  the  level  of  integration  of  the  prototype  can  i  'tively 
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be  related  to  the  type  and  magnitude  of  the  risks  addressed.  Combi¬ 
nations  of  purposes,  objectives,  integration  level,  and  program  phase 
thus  reflect  prototyping  strategies.* 

This  subsection  examines  the  wide  range  of  the  basic  elements  that 
define  a  prototyping  strategy:  goals,  timing,  and  level  of  integration. 
The  importance  of  these  elements  is  that  they  potentially  have  a 
strong  influence  on  other  characteristics  of  development  programs.  A 
wide  range  in  the  basic  elements  would  indicate  a  wide  range  of  pro¬ 
totyping  strategies. 

Figure  4.1  indicates  that  most  of  the  systems  in  the  literature 
database  were  prototyped  at  the  partial  system  level — the  platform 
and  one  or  two  of  the  basic  subsystems.  There  are,  however,  a  signifi¬ 
cant  number  of  full  system  prototypes.  The  subsystem  category  con¬ 
tains  mostly  system  upgrades,  as  we  shall  see  later.  Note  that  this 
category  of  subsystems  differs  in  kind  and  context  from  the  subsys¬ 
tem  category  in  weapon  type.  It  is  fully  possible  that  a  subsystem 
program  prototypes  at  the  full  system  level  (e.g.,  all  key  components 
of  the  subsystem  are  fully  integrated). 

The  program  manager  questionnaire  presents  a  somewhat  different 
picture.  Most  of  the  31  prototyping  programs  in  the  survey  were  pro¬ 
totyped  at  either  the  full  system  level  or  the  subsystem  level.  Only  16 
percent  of  the  respondents  indicated  prototyping  at  the  partial  system 
level. 

Conventional  wisdom  leads  to  the  expectation  that  most  prototyping 
is  done  at  the  partial  system  level.  However,  both  the  literature  re¬ 
view  and  the  survey  indicate  that  a  significant  amount  of  prototyping 
is  done  at  the  full  system  level  using  articles  that  are  apparently  pro¬ 
duction  representative.  It  is  important  to  recognize  that  full  system 
prototyping  does  not  mean  a  complete,  production -ready  article  that 
can  be  deployed  to  operational  forces.  Rather,  it  means  a  system  built 
to  scale  that  has  integrated  those  subsystems  critical  to  system  suc¬ 
cess.  In  fact,  a  possible  explanation  for  full  system  prototyping  is  that 
only  with  full  system  prototyping  can  the  major  integration  problems 
be  identified  and  resolved.  Examples  include  the  Army’s  Army  Heli¬ 
copter  Improvement  Program  (AHIP  OH-58D)  and  Northrop’s  F-20. 


*There  are  obviously  other  associated  components  of  a  prototyping  strategy.  In  fact, 
a  prototyping  strategy  is  simply  a  subset  of  a  much  larger  set  of  general  acquisition 
strategics.  Other  aspects  of  these  strategics  include  contract  type,  concurrency, 
funding  distributions  over  time,  the  number  of  contractors,  fabrication  methods,  and 
design  team  size  and  composition,  among  other  things.  In  part,  the  program  manager 
survey  was  intended  to  collect  some  information  on  these  other  strategy  elements. 
These  data  are  presented  in  a  later  section. 
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Literature  review 
(n  =  143) 


Program  manager 
survey  (n  =  31) 


I  I  Full  system  Subsystem 

I  I  Partial  system  Hi  Other 


Figure  4.1 — Distribution  by  Level  of  Integration 


We  expected  that  full  system  prototypes  would  occur  later  in  a  pro¬ 
gram  and  would  be  relatively  more  expensive  (compared  with  partial 
or  subsystem  prototypes),  with  correspondingly  different  purposes 
and  objectives.  The  next  subsection  examines  this  notion. 

The  program  phase  in  which  the  prototyping  activities  occur  is  also 
important  in  terms  of  defining  prototyping  strategies.  We  would  ex¬ 
pect  differences  in  purpose  and  objectives  across  phases.  Figure  4.2 
shows  the  distribution  of  prototype  phases.  It  is  interesting  that  FSD 
is  dominant  in  both  the  literature  and  survey  data  at  similar  propor¬ 
tions,  as  one  commonly  held  belief  is  that  all  true  prototyping  occurs 
prior  to  FSD.  However,  these  data  suggest  that  prototypes  can  in  fact 
occur  in  any  phase,  the  differences  being  a  function  of  the  purpose 
and  objectives,  and  of  the  relative  risks  involved.  The  category 
“other”  captures  systems  that  are  not  formally  programs  and  may  not 
even  lead  to  a  formal  program.  The  results  on  timing  are  consistent 
with  both  the  survey  and  literature  review  results  on  level  of  integra¬ 
tion. 
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11.8%  9.7%  6.5% 


Literature  review 
(n=  136) 


Program  manager 
survey  (n  =  31) 


"Other"  category  for  literature 
review  includes  nonprograms. 
"Production"  category  for 
survey  refers  to  pilot  facility. 


[  1  Concept 

1  J  Full-scale 

exploration 

development 

1  1  Demonstration/ 

Production 

validation 

IH  Other 

Figure  4.2 — Phase  in  Which  Prototyping  Occurred 


The  most  common  purposes,  both  main  and  secondary,  are  dominated 
by  technology  demonstration  and  system  design,  as  Figure  4.3  shows. 
For  the  literature  data,  a  cross-tabulation  of  these  two  variables 
shows  a  strong  association  between  them.  Technology  demonstration 
prototypes  often  have  either  technology  viability  or  system  design  as 
secondary  purposes,  depending  on  whether  the  program  technical 
risks  are  high  or  low  and  on  the  degree  of  maturity  in  the  technology. 
System  design  prototypes  often  have  technology  demonstration  as  a 
secondary  purpose.  This  combination  implies  a  different  focus  than  if 
the  ordering  of  the  two  purposes  were  reversed.  In  the  former  case, 
the  technology  is  less  mature  with  greater  technical  risk;  in  the  latter 
case,  the  technology  is  more  mature  with  less  technical  risk  associ¬ 
ated  with  it.  The  three  combinations  of  main  and  secondary  purposes 
described  above  account  for  more  than  two-thirds  of  the  total  in  the 
literature  review  data.  These  strong  relationships  indicate  that  our 
taxonomy  does  in  fact  help  identify  prototyping  strategies. 
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Main  Purpose 


Literature  review  .  Program  manager 

(n  =  174)  survey  (n  =  31) 


37.4% 


Literature  review 
(n  =  91) 


Program  manager 
survey  (n  =  24) 


5.5%  8.8% 


4.2% 


Secondary  Purpose 


I  I  Tech,  viability  System  design  IH  Other 

1  I  Technology  demo  Marketing 


Figure  4.3 — Distribution  of  Overall  Purposes 


While  the  proportions  differ  slightly,  the  survey  results  lend  credence 
to  the  overall  pattern  and  distribution  of  main  and  secondary  pur¬ 
poses.  The  relationship  between  purposes  was  less  clear,  however.  A 
program  with  a  main  purpose  of  system  design  had  an  equal  number 
of  both  technology  viability  and  technology  demonstration  protot)rpes, 
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while  programs  with  a  main  purpose  of  technology  demonstration 
more  often  had  system  design  as  a  secondary  purpose.  This  implies 
that  the  relationships  vary  widely  due  to  the  unique  characteristics  of 
weapon  system  development  programs.  It  is  interesting  that  while  no 
programs  indicated  marketing  as  a  main  purpose,  four  programs  did 
indicate  it  as  a  secondary  purpose.  This  is  explained  by  the  observa¬ 
tion  that  the  government  program  managers  responding  to  the  survey 
were  less  likely  to  consider  prototypes  as  marketing  tools  than  was 
private  industry. 

The  frequency  distributions  of  the  three  levels  of  specific  objectives 
(derived  from  the  literature)  are  shown  in  Table  4.1.  Given  that  the 
objectives  are  scaled  in  order  of  relative  importance,  several  interest¬ 
ing  observations  can  be  made.  There  seems  to  be  a  slight  downward 
movement  in  the  relative  frequency  of  particular  objectives  as  we 
move  across  the  levels  of  objectives.  That  is,  prototyping  programs’ 
principal  objectives  have  tended  to  focus  on  the  resolution  of  technical 
risks,  whereas  secondary  objectives  have  often  addressed  develop¬ 
mental  engineering  and  operational  issues.  The  implication  here  is 
that  technical  risk  considerations  tend  to  be  the  primary  focus  of  pro¬ 
totyping  programs  while  operational  considerations  are  of  secondary 
importance.  However,  there  is  considerable  program-to-program 
variation  reflected  in  these  data. 

Table  4.1 

Distribution  of  Specific  Objectives* 


Specific 
Objective  1 

Specinc 
Objective  2 

Specific 
Objective  3 

(No.) 

(%) 

(No.) 

(%) 

(No.) 

(%) 

Experimental 

24 

14.9 

0 

0.0 

0 

0.0 

Exploratory 

22 

13.7 

20 

18.2 

0 

0.0 

Feasibility 

33 

20.5 

26 

23.6 

4 

7.0 

Competitive 

33 

20.5 

2 

1.8 

4 

7.0 

Developmental 

10 

6.2 

26 

23.6 

11 

19.3 

Integration 

1 

0.6 

9 

8.2 

3 

5.3 

Political 

0 

0.0 

2 

1.8 

5 

8.8 

Preproduction 

15 

9.3 

8 

7.3 

5 

8.8 

Missionized 

0 

0.0 

12 

10.9 

14 

24.6 

Operational 

1 

0.6 

4 

3.6 

7 

12.3 

Upgrade 

22 

13.7 

1 

0.9 

4 

7.0 

Total 

161 

100.0 

110 

100.0 

57 

100.0 

'This  is  an  ordinal  scale;  specific  objective  1  is  defined  to  be  more 
important  to  the  program’s  overall  goal  than  either  objective  2  or  3,  but  we 
cannot  know  how  much  more  important. 
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Within  any  given  program,  certain  types  of  objectives  are  associated 
with  certain  other  types.  Common  combinations  include  experimen¬ 
tal  and  exploratory;  competitive  and  feasibility;  and,  exploratory  and 
developmental.  Thus,  there  seem  to  be  strong  associations  between 
particular  types  of  specific  objectives,  implying  that  achievement  of 
certain  objectives  precludes  attainment  of  other  objectives.  These 
common  combinations  again  suggest  that  prototyping  strategies  are 
highly  variable  and  are  oriented  toward  the  specific  needs  and  envi¬ 
ronment  of  a  particular  program 

At  the  macro  level,  the  basic  elements  of  prototyping  strategies — 
goals  (purpose  and  objectives),  timing  (prototyping  phase),  and  level 
of  integration — ^have  varied  widely  in  the  past.  In  both  theory  and 
practice,  prototyping  appears  to  be  a  complex  family  of  strategies, 
rather  than  a  single  approach. 

COMMON  STRATEGIES 

This  subsection  discusses  the  relationships  between  the  basic  ele¬ 
ments  that  define  a  prototyping  strategy:  goals,  timing,  and  level  of 
integration.  We  believe  that  there  are  distinct  relationships  between 
these  aspects  of  prototyping  strategies,  combinations  of  which  define 
a  particular  strategy.  For  instance,  we  might  intuitively  believe  that 
programs  pushing  the  state  of  the  art  in  a  particular  technology 
would  have  low  levels  of  integration,  occur  early  in  a  program  with 
few  prototypes,  and  have  technology  demonstration  as  the  main  pur¬ 
pose  of  the  prototype  phase,  with  perhaps  feasibility  and  developmen¬ 
tal-specific  objectives. 

These  kinds  of  relationships  are  an  integral  part  of  the  conceptual 
framework  discussed  previously.  Under  that  framework,  the  three 
basic  elements  of  a  prototyping  strategy  are  main  purpose,  integra¬ 
tion  level,  and  program  phase,  corresponding  to  the  more  general  el¬ 
ements  of  goals,  integration,  and  timing.  We  would  expect  that  inte¬ 
gration  level  would  vary  across  purposes;  Purposes  corresponding  to 
higher  technical  risks  would  have  lower  integration  levels,  while  more 
mature  technology  would  imply  higher  levels  of  integration.  This  is 
in  fact  what  we  find  when  we  examine  Table  4.2.®  Programs  in  our 
database  having  technological  risk  reduction  as  a  principal  purpose 
have  tended  to  prototype  more  at  the  partial  system  level  earlier  in 


^Similar  results  can  be  observed  in  the  program  manager  responses,  though  the 
numbers  of  programs  in  a  particular  cell  are  quite  small.  See  Table  B.l  for  details. 
^Table  B.6  shows  similar  results  from  the  survey  responses. 
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Table  42 

Elements  of  a  Prototyping  Strategy 


Main  Purpose 


Technology 

Viability 

Technology 

Demonstration 

System 

Design 

Marketing 

ToUl 

Integration  level 

Full  system 

0 

9 

24 

9 

42 

Partial  system 

17 

28 

15 

2 

62 

Subsystem 

1 

6 

13 

1 

21 

Total 

18 

43 

52 

12 

125 

Phase  occurring 

Concept  exploration 

16 

8 

0 

0 

24 

Demonstration/valid. 

0 

15 

7 

0 

22 

Full-scale  develop. 

1 

9 

30 

2 

42 

Other  (nonprogram) 

5 

6 

3 

11 

25 

Total 

22 

38 

40 

13 

113 

the  acquisition  process,  whereas  those  with  more  of  a  system  or  mar¬ 
keting  purpose  have  more  frequently  prototyped  at  the  full  system 
level  later  in  the  development  cycle. 

Similarly,  there  is  a  strong  relationship  between  the  phase  in  which 
the  prototyping  activities  occur  and  the  level  of  integration  of  the  pro¬ 
totype  article.*  Full  system  prototypes  are  more  common  during 
full-scale  development,  while  partial  system  prototyping  occurs  more 
often  in  pre-FSD  phases,  especially  in  the  concept  phase  when  perfor¬ 
mance  requirements  are  being  formulated. 

Associations  of  purposes  and  objectives  also  provide  a  strong  indica¬ 
tion  that  some  prototyping  strategies  are  more  common  than  others. 
As  the  technology  matures,  objectives  focus  less  on  pure  technology 
application  objectives  (experimental,  exploratory,  feasibility),  and 
more  on  demonstration  of  performance  and  military  utility.  The  same 
pattern  is  found  in  the  survey  results.®  In  particular,  programs  with 
main  purposes  of  technology  viability  and  technology  demonstration 
have  few  objectives  at  any  level  that  involve  the  more  mature  end  of 
the  spectrum:  preproduction,  missionized,  operational,  and  upgrade 
objectives.  On  the  other  hand,  system  design  main  purposes  com¬ 
monly  have  developmental  as  the  first-order  objective  and  integration 

*See  Tables  B.4  and  B.5  for  details. 

^See  Tables  B.2  and  B.3  for  support  from  the  literature  and  survey  data. 
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as  the  second-order  objective.  Few  programs  of  any  kind  admit  to 
having  had  political  objectives. 

While  there  appear  to  be  wide  variations  in  prototyping  strategies, 
deflned  as  combinations  of  the  three  basic  strategy  elements  (goals, 
integration,  timing),  several  strong  relationships  emerge.  Partial  sys¬ 
tem  prototyping  appears  to  be  more  often  associated  with  technology 
demonstration  purposes  and  occurs  earlier  in  a  program,  while  full 
systems  are  more  often  associated  with  system  design  purposes  and 
occur  during  FSD.  In  fact,  a  few  strategies  seem  to  have  been  more 
common  in  past  prototyping  experience,  at  least  at  this  aggregate 
level.  As  we  will  observe  in  a  later  section,  however,  other  program¬ 
matic  characteristics  vary  widely  across  these  few  common  strategies, 
implying  that  a  great  deal  of  tailoring  goes  on  at  the  detailed  level. 

VARIATIONS  IN  THE  BASICS:  SYSTEM  TYPE  AND  AGENCY 

This  section  examines  whether  the  common  protot)T)ing  strategies, 
and  the  relationships  between  the  basic  elements  of  prototyping 
strategies,  vary  with  the  type  of  weapon  system  and/or  the  organiza¬ 
tion  responsible  for  funding  and  management. 

We  might  expect  variations  in  prototyping  strategies  across  weapon 
types  due  to  differences  in  technology,  technical  difficulty,  or  devel¬ 
opment  process.  In  other  words,  some  weapon  types  may  be  more 
commonly  prototyped  in  certain  phases,  at  particular  levels  of  inte¬ 
gration,  and  with  certain  main  purposes.  Table  4.3  explores  some  of 
these  relationships  using  the  literature  review  database.^  For  fixed- 
wing  aircraft,  there  does  not  seem  to  be  much  difference  across  pur¬ 
poses.  Helicopters,  however,  seem  to  be  more  commonly  system 
design  prototypes,  while  land  vehicles  are  more  or  less  equally  dis¬ 
tributed  between  technology  demonstration  and  system  design  pur¬ 
poses.  Missiles  and  subsystems  are  more  often  associated  with  a 
technology  demonstration  purpose.  Similar  patterns  are  observed 
when  examining  the  associations  of  weapon  type  and  the  more  de¬ 
tailed  specific  objective  level. 

With  the  experimental  V/STOLs  (Vertical/Short  Takeoff  and  Land¬ 
ings)  of  the  1960s  biasing  the  aircraft  category,  no  real  pattern 
emerges  between  weapon  type  and  the  timing  of  the  prototyping  ef¬ 
fort,  as  measured  by  the  program  phase  in  which  the  prototype  activ- 


^Whcn  divided  into  so  many  categories,  the  program  manager  survey  data  show  no 
dominant  patterns,  but  they  do  not  refute  these  other  results.  Table  B.7  provides  the 
details. 


Table  4^ 

Strategies  by  Weapon  IVP® 
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Weapon  Type 

Aircraft 

Heli¬ 

copter 

Land 

Vehicle 

Missile 

Sub¬ 

system 

Other 

Total 

Main  purpose 
Technology 
viability 

18 

2 

1 

0 

4 

1 

26 

Technology 

demonstration 

16 

6 

10 

17 

11 

4 

64 

System  design 

29 

11 

14 

8 

3 

0 

65 

Marketing 

5 

1 

6 

4 

2 

1 

19 

Total 

68 

20 

31 

29 

20 

6 

174 

Phase  occurring 
Concept 
exploration 

17 

0 

5 

2 

3 

1 

28 

Demonstration/val¬ 

idation 

6 

4 

2 

9 

5 

2 

28 

Full-scale 

development 

28 

7 

5 

10 

6 

0 

56 

Other 

(nonprogram) 

8 

6 

3 

6 

2 

1 

26 

Total 

59 

17 

15 

27 

16 

4 

138 

Integration  level 

Full  system 

15 

1 

12 

11 

3 

3 

45 

Partial  system 

38 

10 

9 

3 

3 

2 

65 

Subsystem 

13 

4 

1 

7 

6 

2 

33 

Total 

66 

15 

22 

21 

12 

7 

143 

ities  take  place.  There  is  high  variability  in  the  timing  of  prototyping 
activities  across  weapon  types. 

Integration  level  shows  only  a  slightly  better  association  with  weapon 
type.  Proportionately  more  aircraft  and  helicopters  are  partial  sys¬ 
tems.  Land  vehicles  and  missiles  tend  to  be  full  systems,  and  subsys¬ 
tem  prototypes  show  no  significant  distinction. 

Table  4.3  gives  the  impression  that  prototyping  strategies  vary  sub¬ 
stantially  across  weapon  types,  with  few  suggestions  of  strong  rela¬ 
tionships.  The  implication  is  that  any  weapon  type  can  incorporate 
any  prototyping  strategy  deemed  appropriate  by  the  program’s  man¬ 
agers.  There  does  not  appear  to  be  a  dominant  strategy  in  any  cate¬ 
gory  of  weapon  system.  A  more  detailed  breakdown  examining  the 
relationships  between  the  elements  of  a  prototyping  strategy  and  par- 


50 


ticular  weapon  types  (cross-tabulations  for  each  system  type  one  at  a 
time)  supports  this  view.  The  implication  is  that  something  else  dom¬ 
inates  the  appropriateness  of  prototyping  strategies.  It  has  already 
been  suggested  that  the  type  and  nature  of  the  risks  and  the  relative 
level  of  system  maturity  fill  this  role.  To  the  extent  that  these  risks 
are  common  across  weapon  system  types,  the  appropriateness  of  a 
given  strategy  does  not  vary. 

We  might  also  expect  variations  in  prototyping  strategies  across 
management  agencies,  reflecting  differences  in  management  style 
and  emphasis.  The  associations  between  management  agency  and 
the  elements  of  a  prototyping  strategy  are  only  slightly  stronger  than 
for  weapon  types.'^  Table  4.4  illustrates  this.  In  terms  of  the  main 
purpose  of  the  prototype.  Air  Force  and  Army  programs  are  evenly 
distributed  across  technology  demonstration  and  system  design  pur¬ 
poses,  while  the  Navy  and  foreign  government  programs  seem  to  fo¬ 
cus  mostly  on  system  design  considerations.  Joint  programs  between 
U.S.  services  are  dominated  by  technology  demonstration  prototypes, 
perhaps  reflecting  the  fact  that  these  are  mostly  complicated  electron¬ 
ics-based  programs  advancing  the  state  of  the  art.  Private  ventures, 
both  foreign  and  domestic,  are  generally  marketing  prototypes,  re¬ 
flecting  the  orientation  of  most  private  industry.  At  the  more  detailed 
specific  objective  level,  the  Air  Force  and  the  Army  both  have  sub¬ 
stantial  proportions  of  competitive  prototypes  as  a  principal  objective, 
while  Navy  programs  are  more  evenly  distributed.  It  should  be  noted 
that  data  availability  and  quality  were  poor  for  the  Navy  programs. 

There  appears  to  be  some  variation  in  the  phase  in  which  the  proto¬ 
typing  activities  occur.  Both  Air  Force  and  Navy  programs  are  more 
commonly  prototyped  during  FSD,  though  a  substantial  number  of 
Air  Force  programs  are  also  prototyped  during  demonstration/ 
validation.  Army  programs  are  the  reverse  of  Air  Force  programs, 
with  most  in  demonstration/validation  but  a  substantial  number  in 
FSD.  Again  reflecting  the  private-venture  nature  of  marketing  proto¬ 
types,  privately  funded  prototypes  tend  to  be  outside  the  normal  pro¬ 
gram  phase  structure. 

Integration  level  shows  some  association  with  management  agency  as 
well.  While  Air  Force  prototypes  are  well  distributed  across  levels  of 
integration.  Navy  programs  have  proportionately  more  subsystem 


Again,  the  program  manager  survey  data  do  not  show  any  dominant  patterns 
when  divided  into  so  many  cells.  The  exception  is  that  Army  programs  appear  to  be 
more  often  lull  ^stem  prototypes  with  system  design  purposes  done  during  FSD.  See 
Table  B.8  for  details. 
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prototypes,  reflecting  the  current  trend  toward  system  upgrades. 
Army  programs,  however,  are  balanced  between  full  and  partial  sys¬ 
tems.  Again  reflecting  the  nature  of  marketing  prototypes,  private 
ventures  tend  to  be  fully  integrated  systems. 

Table  4.4  provides  some  slight  support  for  the  notion  that  prototype 
strategies  vary  across  management  agencies.  This  probably  reflects 
differences  in  management  style  and  emphasis  rather  than  differ¬ 
ences  in  prototype  or  weapon  system  characteristics.  With  the  excep¬ 
tion  of  private  ventures,  there  is  no  intuitive  reason  why  any  particu¬ 
lar  prototyping  strategy  cannot  be  applied  by  any  management 
agency. 

It  is  entirely  possible  that  the  macro-level  view  adopted  here — exam¬ 
ining  variations  in  prototyping  goals,  timing,  and  integration — may 
obscure  differences  in  prototyping  strategies  across  weapon  system 
type  and  management  agency.  In  theory,  these  differences  would  be 
more  identifiable  at  a  more  detailed  level  of  analysis  and  may  also 
show  up  in  variations  in  programmatic  characteristics  associated 
with  the  basic  elements  of  prototyping  strategies.  The  next  section 
begins  to  examine  some  of  these  additional  aspects  of  prototyping.^ 

VARIATIONS  IN  ASSOCIATED  STRATEGY 
CHARACTERISTICS 

Under  the  conceptual  framework  developed  as  part  of  this  research, 
the  three  basic  prototyping  strategy  elements — goal,  timing,  and  inte¬ 
gration — should  in  part  determine  a  myriad  of  other  programmatic 
characteristics.  For  instance,  we  might  expect  that  main  purposes 
would  be  associated  with  perceived  or  expected  benefits,  or  that  re¬ 
quirements  specification  and"  contract  type  would  be  associated  with 
both  the  main  purpose  and  the  timing  of  the  prototyping  phase. 
Based  mostly  on  the  program  manager  survey,  this  subsection  ex¬ 
plores  some  of  these  relationships. 

Given  an  identified  set  of  risks  and  uncertainties  associated  with  a 
particular  program,  we  would  expect  that  the  prototyping  strategy 
chosen  would  reflect  the  perceived  benefits  of  prototyping.  Table  4.5 
shows  the  perceived  benefits  of  prototyping  for  the  31  survey  re¬ 
sponses.  These  factors  vary  widely,  corresponding  with  the  high 


^Unfortunately,  the  program  manager  survey  data  will  not  support  a  statistical 
analysis  at  that  level  of  detail.  There  are  too  few  observations  in  any  category  of 
weapon  system  or  management  agency  to  draw  credible  conclusions. 


Table  4^ 

Perceived  Benefits  of  Prototyping  (Survey  Results) 
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Main  Purpose 

Technology 

Technology 

System 

BeneHt  of  Prototyping 

Viability 

Demo 

Design  Other 

Prove  adequacy  of  design 

1 

5 

Validate  performance 

4 

Validate  requirement 

1 

Early  test,  fix,  test 

Reduce  risk 

1 

4 

2 

Technology  demonstration 

2 

2 

2 

Competition 

Pro%ride  info  for  trade-off 

1 

1 

Other 

1 

4  1 

variation  in  prototyping  strategies  observed  earlier.  Perceived  bene¬ 
fits  range  from  proving  the  adequacy  of  the  design  and  demonstrating 
technology  to  validating  system  performance  and  requirements. 
There  is  some  relationship  to  main  purpose  (which  is  directly  related 
to  program  risk  under  the  conceptual  framework),  with  more  risk-ori¬ 
ented  and  information-generating  benefits  associated  with  technology 
demonstration  purposes,  while  system  design  prototypes  are  more  of¬ 
ten  associated  with  validation  benefits.  Note  that  there  is  significant 
agp'eement  between  the  survey  respondents’  expectations  of  benefits 
(Table  4.5)  and  more  general  perceived  prototyping  benefits  (Table 
2.1).9 

The  survey  also  yielded  some  insight  into  other  aspects  of  prototyping 
strategies.  Conventional  wisdom  suggests  that  prototyping  works 
best,  and  is  in  fact  more  common,  when  single  point  requirements  are 
not  specified  in  detail  and  when  they  can  be  modified  as  a  result  of 
testing.  About  half  of  the  programs  in  the  survey  had  requirements 
specified  as  part  of  the  prototype  contract,  not  much  different  than 
nonprototyping  programs,  with  the  other  half  incorporating  more  flex¬ 
ible  contracting  strategies:  best  effort  and  performance  ranges  and 
goals.  Interestingly,  only  about  half  of  the  programs  modified  the  re¬ 
quirements  as  a  result  of  prototype  testing;  those  requirements  were 
specified  in  the  contract. Rather  than  substantiating  conventional 
wisdom,  the  survey  results  suggest  high  variability  regarding  re- 


^nfortunately,  this  database  cannot  directly  address  the  more  important  issue: 
whether  perceived  benefits  were  actually  obtained. 
lOSee  Table  B.ll  for  details. 
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quirement  specification  and  the  flexibility  to  modify  those  require¬ 
ments. 

Similarly,  the  actual  contract  types  for  prototyping  programs  show 
some  counterintuitive  results:  40  percent  of  the  respondents  indi¬ 
cated  that  the  prototype  phase  was  conducted  under  a  fixed-price  con¬ 
tract,  a  strategy  that  makes  sense  only  if  the  contract  is  a  best  effort 
type.  Table  4.6  shows  the  relationship  between  contract  type  and  the 
way  in  which  requirements  were  specified.  There  is  a  high  variabil¬ 
ity,  but  only  three  of  ten  programs  with  fixed-price  prototyping  con¬ 
tracts  specified  requirements  in  the  contract.  The  remainder  were 
best  effort  or  other  flexible  types  of  performance  specification.  As  we 
would  expect,  contract  types  show  less  variability  in  the  FSD  phase, 
and  production  contracts  were  almost  entirely  fixed  price.'* 

Interestingly,  contract  type  during  the  prototyping  phase  does  not 
seem  to  be  strongly  associated  with  any  of  the  basic  prototyping 
strategy  elements.  There  is  a  wide  variation  in  contract  types  across 
all  categories  of  main  purpose,  prototyping  phase,  and  integration 
level. '2  However,  there  is  a  fair  correspondence  between  contract 
tjTpe  in  the  prototyping  and  FSD  phases:  firm  fixed-price  and  cost 
plus  incentive  prototype  contracts  tend  to  be  associated  with  similar 
contract  types  during  FSD.'^ 

Other  aspects  of  prototyping  strategies  show  some  variations  and  re¬ 
lationships  across  program  phases.  Table  4.7  suggests  that  while 


Table  4.6 

Contracting  and  Requirements  Specification:  Prototyping  Phase 


Type  of  Contract 

Form  of  Requirement 

FFP 

CPI 

CPIw/c 

CPFF 

Other 

Total 

Contract  specific 

3 

4 

1 

1 

3 

12 

Performance  goal 

2 

2 

2 

1 

0 

7 

Performance  range 

1 

0 

0 

1 

0 

2 

Best  effort 

2 

0 

0 

0 

0 

2 

Other 

2 

0 

0 

0 

0 

2 

Total 

10 

6 

3 

3 

3 

25 

NOTES:  FFP  =  firm  fixed  price;  CPI  =  cost  plus  incentive;  CPIw/c  =  cost  plus 
incentive  w/ceiling;  CPFF  =  cost  plus  fixed  fee. 


"See  Table  B.IO  for  details. 
12Sce  Table  B.12. 

13See  Table  B.9. 


Table  4.7 

Nature  of  Technical  Advance  and  Timing 
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Prototyping  Phase 

Concept 

Explor. 

Dem./Val. 

FSD  Production 

Other 

Technical  advance 

Evolutionary 

1 

7 

12  1 

1 

Revolutionary 

1 

4 

2 

2 

When  planned 

Before 

2 

9 

11 

2 

After 

2 

2  1 

1 

n  =  32  for  Technical  advance;  n  =  30  for  When  planned. 


most  programs  consider  themselves  to  be  evolutionary  in  nature,  the 
revolutionary  programs  more  often  are  prototyped  earlier  in  a  devel¬ 
opment  program.  Given  the  nature  of  the  risks  involved,  that  out¬ 
come  is  what  we  would  expect:  Higher  risk  programs  are  prototyped 
sooner,  most  often  to  demonstrate  technical  feasibility.  It  is  also  in¬ 
teresting  that  most  prototype  programs  are  planned  to  include  proto¬ 
typing  from  the  outset,  which  seems  to  be  a  more  efficient  acquisition 
strategy  than  making  changes  to  an  ongoing  development  program. 
The  hierarchical  level  of  the  official  who  makes  the  decision  to  proto¬ 
type  was  equally  distributed  across  the  spectrum  from  the  USD(A) 
level  to  the  program  manager.  On  average,  there  were  two  organiza¬ 
tional  decisionmaking  layers  between  the  program  manager  and  the 
top  decision  maker  for  the  prototype,  FSD,  and  production  phases. 

Though  we  might  expect  that  the  number  of  prototypes  fabricated  and 
tested  would  be  related  to  the  elements  of  prototyping  strategy,  this 
analysis  finds  no  evidence  in  support  of  this  notion.  Most  programs 
had  relatively  few  prototypes,  with  no  apparent  relationship  with 
purpose,  integration  level,  or  phase:  68  percent  of  the  programs  from 
the  literature  review  data  have  five  or  less  prototypes,  relatively 
evenly  distributed  across  main  purposes  and  integration  levels.  The 
number  of  prototypes  might  instead  be  related  to  the  cost  or  time 
available  for  prototyping  activities.  That  notion  is  explored  in  a  later 
section. 

Consistent  with  the  observation  that  prototyping  strategies  vary 
widely,  programmatic  characteristics  also  vary  widely.  In  many 
cases,  aspects  of  programs  such  as  contract  type,  the  way  in  which  re¬ 
quirements  are  specified,  and  relative  technological  advance  appear 
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to  be  related  not  only  to  the  goals,  timing,  and  integration  dimensions 
of  prototyping  strategies,  but  also  to  each  other. 

An  interesting  question  that  arises  at  this  point  is  whether  prototyp¬ 
ing  strategies  determine  programmatic  characteristics  or  whether 
those  characteristics  determine  that  strategy.  Our  conceptual  frame¬ 
work  hypothesizes  that  the  programmatic  characteristics  flow  from 
the  basic  strategy  elements,  but  this  cannot  be  proved  definitively;  a 
credible  argument  can  be  made  for  either  relationship.  The  evidence 
presented  in  this  section  suggests  only  that  there  is  some  strong 
interaction. 

PROTOTYPING  STRATEGIES  OVER  TIME 

We  have  already  seen  that  both  the  literature  review  and  program 
manager  survey  databases  used  in  this  analysis  are  somewhat  biased 
because  data  were  more  readily  available  on  more  programs  in  more 
recent  years  (see  Figures  3.4  and  3.5).  Recognizing  that  there  could 
have  been  more  prototype  programs  in  earlier  years,  we  can  still  look 
for  trends  in  the  data. 

The  basic  question  is  whether  the  nature  of  prototyping  has  changed 
over  time.  We  approached  this  by  examining  the  trends  in  the  basic 
elements  that  comprise  prototyping  strategies.  Table  4.8  shows  the 
main  purposes  of  prototyping  activities  by  five-year  periods  based  on 
the  year  the  program  started.  The  bias  mentioned  earlier  is  reflected 
in  the  last  column.  The  data  show  that  technology  viability  was  the 
most  frequent  purpose  prior  to  1965,  while  technology  demonstration 
and  system  design  purposes  were  more  common  in  later  years.  Notice 

Table  4A 

Changes  in  Prototyping  Purpose  over  Time 
Main  Purpose 


Time  Period 

Technology 

Viability 

Technology 

Demonstrator 

System 

Design 

Marketing 

Total 

1955-59 

3 

2 

0 

0 

5 

1960-64 

8 

2 

6 

1 

17 

1965-69 

2 

3 

7 

1 

13 

1970-74 

1 

12 

9 

0 

22 

1975-79 

1 

4 

8 

2 

15 

1980-84 

3 

13 

12 

7 

35 

1985+ 

2 

6 

7 

2 

17 

Total 

20 

42 

49 

13 

124 
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also  that  there  were  substantially  more  marketing  prototypes,  mostly 
private  ventures,  in  the  early  1980s.  While  admittedly  biased  toward 
more  recent  programs,  the  data  do  provide  some  slight  evidence  that 
the  main  purposes  of  prototypes  have  changed  somewhat,  leaning  to¬ 
ward  proof  of  performance  and  operational  utility  and  away  from 
demonstrations  of  technological  feasibility.  A  similar  pattern 
emerges  in  the  specific  objectives.*'* 

Table  4.9  shows  a  fairly  strong  trend  with  respect  to  the  program 
phase  in  which  prototyping  activities  occur.  Earlier  periods  appear 
to  have  had  more  concept  exploration  prototypes,  while  in  the  early 
1970s,  there  were  more  prototypes  occurring  in  demonstration/ 
validation.  That  result  is  consistent  with  the  trends  in  main  purpose; 
technology  viability/concept  exploration  phase  and  technology 
demonstration/demonstration  phase  combinations  are  some  of  the 
more  common  strategies.  Further,  the  trend  in  the  timing  of  proto¬ 
typing  activities  may  be  a  result  of  Deputy  Secretary  Packard’s  push 
for  “advanced  development”  prototypes  in  the  early  1970s.  The  ac¬ 
quisition  programs  of  the  late  1970s  and  early  1980s  had  more  proto¬ 
typing  during  the  FSD  phase.  Reasons  for  these  patterns  might  in¬ 
clude  the  recent  trend  toward  upgrading,  which  is  often  an  FSD  ef¬ 
fort,  or  use  of  more  mature  technology  in  recent  systems.  It  is  also 
noteworthy  that  there  were  more  prototypes  outside  of  the  normal 
service  programs  in  more  recent  years.  These  are  the  marketing  pro¬ 
totypes  that  we  saw  in  Table  4.8.  The  trend  in  marketing  prototypes 


Table  4.9 

Trends  in  Phase  When  Prototyping  Occurred 


Time  Period 

Acquisition  Phase 

Total 

Concept 

Exploration 

Demonstration/ 

Validation 

Full-Scale 

Development 

Other 

(Nonprogram) 

1955-69 

2 

1 

2 

1 

6 

1960-64 

8 

0 

7 

0 

15 

1965-69 

2 

3 

7 

2 

14 

1970-74 

2 

13 

4 

2 

21 

1975-79 

0 

3 

10 

3 

16 

1980-84 

3 

4 

17 

7 

31 

1985-f 

2 

1 

3 

2 

8 

Total 

19 

25 

50 

17 

111 

**The  survey  data,  covering  the  period  1970  through  1989,  provide  no  evidence  of  a 
change  in  main  purposes  over  time.  There  appears  to  be  no  significant  difference 
between  pre-  and  post- 1980  programs.  See  Table  B.15. 
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can  perhaps  be  explained  by  the  encouragement  of  private  investment 
in  weapon  system  development  in  recent  years.*® 

Examination  of  Table  4.10  reveals  no  real  pattern  in  integration  level 
over  time.  Though  it  appears  that  fully  integrated  prototypes  are 
more  common  in  recent  years,  this  contradicts  conventional  wisdom 
that  prototyping  costs  have  been  increasing.  Full  system  prototyping 
is  obviously  more  expensive  than  either  partial  system  or  subsystem 
prototyping.  However,  it  is  consistent  with  the  notion  that  system  in¬ 
tegration  is  becoming  a  major  technical  challenge  in  weapon  system 
development.  Full  system  prototypes  are  needed  to  resolve  many  in¬ 
tegration  risks.  It  is  interesting  that  the  frequency  of  subsystem  pro¬ 
totyping  has  remained  relatively  constant  over  time.  Given  the  in¬ 
creasing  costs  of  systems,  the  fact  that  much  of  this  increase  is  due  to 
incorporation  of  greater  amounts  of  more  complex  electronics,  and  the 
trend  toward  integration  of  all  these  electronics  as  one  of  the  major 
technical  challenges  in  weapons  development,  we  might  have  ex¬ 
pected  to  see  an  increasing  trend  in  subsystem  prototyping.  The 
cyclical  pattern  of  partial  systems  remains  unexplained.*® 

The  program  manager  survey  data  allowed  an  analysis  of  trends  in 
some  of  the  programmatic  characteristics  associated  with  prototyping 
strategies.  Results  indicate  no  identifiable  change  over  time  in  con¬ 
tract  types,  requirements  specification,  decision  making,  etc.  In  part, 
the  large  variation  in  the  data  and  the  few  observations  in  any  par¬ 
ticular  grouping  explain  this  result. 


Table  4.10 

Trends  in  Integration  Level  of  Prototypes 


Time  Period 

Integration  Level 

Full  System 

Partial  System 

Subsystem 

Total 

1955-59 

2 

4 

0 

6 

1960-64 

4 

12 

0 

16 

1965-69 

1 

6 

4 

11 

1970-74 

3 

12 

5 

20 

1975-79 

6 

4 

5 

15 

1980-84 

10 

12 

5 

27 

1985-f 

5 

6 

2 

13 

Total 

31 

56 

21 

108 

*®Table  B.16  supports  these  trends  using  the  program  manager  survey  data. 

*®Table  B.17  shows  some  slightly  difTerent  results  from  the  survey.  Subsystem 
prototypes  are  considerably  more  common  in  more  recent  years  for  this  sample  of 
programs,  while  full  system  prototypes  have  decreased  in  frequency. 


5.  PROTOTYPING  AND  PROGRAM  OUTCOMES 


Prototyping  is  thought  to  be  valuable  as  an  acquisition  strategy  be¬ 
cause  it  can  potentially  contribute  to  good  program  outcomes,  as  mea¬ 
sured  by  cost,  schedule,  and  performance.  If  a  prototyping  strategy 
successfully  reduces  program  risks,  then  cost,  schedule,  and  perfor¬ 
mance  outcomes  should  be  improved.  In  attempting  to  demonstrate 
this  contribution,  however,  we  encountered  several  problems. 

One  fundamental  problem  is  that  while  we  can  examine  prototyping 
programs,  or  compare  the  outcomes  of  prototyping  and  nonprototyp¬ 
ing  programs,  the  outcome  for  the  same  program  with  and  without 
prototyping  can  never  be  known.  In  other  words,  we  can  only  hypoth¬ 
esize  about  the  difference  in  outcome  for  a  program  in  which  prototyp¬ 
ing  was  considered  as  a  strategy  and  rejected.  There  is  a  related 
problem  of  confounding  variables.  Many  factors  affect  program  cost, 
schedule,  and  performance  outcomes,  and  it  is  not  often  possible  to 
separate  these  effects  from  each  other.  Hence,  we  can  never  defini¬ 
tively  know  the  relative  contribution  of  prototyping  to  program  out¬ 
comes.  Additionally,  whether  or  not  we  should  expect  prototyping  to 
improve  cost,  schedule,  and  performance  outcomes  (e.g.,  whether 
baseline  estimates  under  prototyping  will  more  accurately  reflect  ac¬ 
tual  outcomes)  is  a  function  of  both  the  timing  of  information  avail¬ 
ability  resulting  from  prototyping  activities  and  the  willingness  of 
decision  makers  to  incorporate  that  information  in  establishing  or  at¬ 
tempting  to  meet  those  baselines. 

An  important  conceptual  problem  is  that  cost,  schedule  and  perfor¬ 
mance  outcomes  do  not  measure  what  we  are  really  interested  in:  the 
cost-effectiveness  of  a  system  in  operational  use.  Unfortunately,  we 
cannot  measure  the  effectiveness  of  a  system  in  operational  use,  or 
the  cost-effectiveness  of  an  acquisition  strategy  or  program  in  deliver¬ 
ing  that  system  for  use,  in  a  meaningful  way.  Thus,  we  are  left  with 
examining  cost,  schedule,  and  performance  outcomes  as  the  only  rea¬ 
sonably  objective  measures  of  program  “success”  available. 

The  observed  effect  of  prototyping  activities  on  program  outcomes 
might  look  either  of  two  ways.  First,  information  gained  through 
early  prototyping  can  be  used  to  improve  the  accuracy  of  our  baseline 
estimates  of  cost,  schedule,  and  performance. '  We  would  expect  that 


^These  three  baselines  are  generally  established  at  the  Milestone  I,  II,  and  Ilia 
decision  points  in  the  developntent  process. 
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the  information  generated  through  prototyping  activities  would  re¬ 
duce  program  risk  by  allowing  managers  to  anticipate  and  plan  for 
technical  problems.  Thus,  a  protot3^ing  program  should  incur  lower 
cost  growth  and  schedule  slip  as  compared  to  nonprototyping  pro¬ 
grams,  assuming  that  this  information  was  used  in  determining  the 
progp’am  cost  and  schedule  baselines  against  which  such  outcomes  are 
measured. 

Second,  the  effect  of  prototyping  may  look  different  if  the  information 
generated  is  not  used  in  formulating  program  baselines,  however. 
For  instance,  one  type  of  successful  prototyping  activity  would  un¬ 
cover  previously  unidentified  technical  problems  affecting  estimates 
of  cost,  schedule,  and  performance.  In  fact,  this  is  arguably  one  of  the 
more  important  functions  of  prototyping,  since  it  directly  addresses 
risk  in  the  program.  From  a  programmatic  perspective,  however,  we 
might  observe  the  result  as  cost  growth  and/or  schedule  slip  mea¬ 
sured  from  a  baseline  that  does  not  include  the  information  generated 
through  prototyping;  hence,  we  really  cannot  tell  whether  greater  or 
lesser  cost  growth  and  schedule  slip  should  be  associated  with  proto¬ 
typing.  If  the  cost  and  schedule  growth  was  measured  from  a  base¬ 
line  calculated  prior  to  the  prototyping  activity,  then  growth  would  be 
expected.  If  the  baselines  were  determined  after  the  prototyping  ac¬ 
tivities,  then  we  might  expect  lower  growth.  It  is  the  timing  and  use 
of  information  generated  during  prototyping  activities  that  is  the  ba¬ 
sis  of  the  effect  of  prototyping  on  program  outcomes. 

Given  the  difficulties  involved,  it  is  not  a  surprise  that  the  evidence 
linking  prototyping  and  program  outcomes  is  fairly  weak:  the  effect  of 
prototyping  on  program  outcomes  is  ambiguous  due  to  confounding 
variables.  In  general  we  do  not  know  enough  about  the  specific  infor¬ 
mation  generated  and  how  it  was  used  to  establish  a  relationship  be¬ 
tween  prototyping  and  program  outcomes  and  identify  a  fundamental 
difference  between  prototyping  and  nonprototyping  program  out¬ 
comes.  Thus,  the  results  presented  here  may  reflect  the  limitations  of 
the  analysis  rather  than  any  relationship  between  prototyping  and 
program  outcomes.  Large  sample  studies  cannot  control  for  all  the 
factors  that  might  affect  outcomes.  However,  case  study  approaches 
have  been  more  successful  in  gaining  insight  into  the  value  of  proto¬ 
typing  for  specific  programs.^  Nonetheless,  available  data  from  this 


^See  G.  K.  Smith  et  al.,  The  Use  of  Prototypes  in  Weapon  System  Development, 
RAND,  R-2345-AF,  March  1981,  and  Mark  A.  Lorell  and  D.  HolTman,  The  Use  of 
Prototypes  in  Selected  Foreign  Fighter  Aircraft  Development  Programs,  RAND,  R-3687- 
P&L,  &ptcmber  1989. 
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and  other  RAND  research  do  offer  some  insight  into  the  relationship 
between  prototyping  and  program  outcomes. 

PROTOTYPING,  PROGRAM  DURATIONS,  AND 
SCHEDULE  SLIP 

Proponents  of  prototyping  believe  that  prototyping  does  not  increase 
actual  progp-am  duration,  while  opponents  believe  that  it  does.  If 
prototyping  had  no  net  effect  on  program  duration,  then  we  might  ex¬ 
pect  to  find  slightly  longer  plans,  on  average,  in  anticipation  of  future 
slip,  but  similar  actual  program  durations.  Figure  5.1  shows  the  ac¬ 
tual  program  duration  for  61  aerospace  programs,  measured  as  the 
time  from  program  start  (Milestone  I  or  equivalent)  to  first  opera¬ 
tional  delivery.  Programs  meeting  the  narrow  definition  of  prototyp¬ 
ing  are  shown.  Several  interesting  observations  can  be  drawn.  First, 
there  is  a  high  degree  of  variation  in  actual  program  durations,  both 
over  time,  between  prototyping  and  nonprototyping  programs,  and 
within  these  categories.  Second,  prototyping  programs  do  appear  to 
take  approximately  12  months  longer  to  reach  first  operational  deliv¬ 
ery  (significant  at  the  10  percent  level).  Table  5.1  provides  some  ad¬ 
ditional  detail  supporting  this.  Planned  program  durations  are  in 
fact  slightly  longer  in  prototyping  programs  than  in  nonprototyping 
programs  (as  expected),  but  subsequent  schedule  slip  is  also  greater, 


1960  1964  1968  1972  1976  1980  1984 

Year  of  Milestone  I 


SOURCE:  Selected  Acquisition  RefMrts  as  ol  December  1988. 

NOTE:  T-lesI  of  difference  between  means  indicates  that  mean  for  prototypes 
is  different  from  nonprofotypes  at  the  10  percent  significance  level. 

Figure  5.1 — Prototyping  and  Actual  Program  Duration 
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Table  5.1 

Duration  and  Slip  Dififerencea  Between  Protot3rping  and 
Nonprototyping  Programa 
(meaaured  in  montha  firom  Mileatone  I  to  firat 
operational  deliveiy) 


Prototyping 

Nonprototyping 

All  programs 

Planned  length 

88.4 

81.0 

85.3 

Pre-FSD  slip 

2.7 

-0.44 

1.3 

Slip  in  PSD 

12.3 

5.1 

9.1 

Total  program  slip 

13.7 

6.2 

9.8 

Actual  duration 

102.4 

91.0 

97.1 

SOURCE:  Data  derived  from  Selected  Acquisition  Reports  through 
December  1988. 


on  average.  Hence,  prototyping  programs  tend  to  have  slightly  longer 
actual  program  durations  than  nonprototyping  programs.  Again,  care 
is  required  in  interpreting  this  result.  It  may  be  that  the  schedule 
slip  experienced  in  prototyping  programs  reflects  a  successful  proto¬ 
typing  activity;  problems  were  identified  and  time  was  taken  to  re¬ 
solve  them,  thus  improving  the  quality  of  the  first  operational  unit 
delivered. 

A  recent  study  examining  factors  affecting  acquisition  program  dura¬ 
tion  provides  inconclusive  evidence  that  prototyping  is  related  to  ei¬ 
ther  program  length  or  schedule  slip.®  Of  the  ten  programs  examined 
in  some  detail,  eight  contained  distinct  prototyping  phases.  The  vari¬ 
ation  in  both  the  planned  and  actual  time  from  program  start  to  first 
operational  delivery  is  high.  Similarly,  the  schedule  slip  experienced 
by  these  programs  is  highly  variable,  ranging  from  0  to  about  80  per¬ 
cent  of  the  original  plans. 

A  broader  view  of  the  relationship  between  prototyping  and  schedule 
outcomes  is  provided  by  Figure  5.2,  which  shows  the  schedule  slip 
(measured  as  the  slip  in  first  operational  delivery)  for  prototyping  and 
nonprototyping  programs.  The  data  indicate  that  on  average,  proto¬ 
typing  programs  incur  more  slip  in  first  delivery  (see  Table  5.1):  13.7 
months  versus  6.2  months  for  prototyping  and  nonprototyping  pro¬ 
grams  respectively.  However,  there  is  considerable  program-to- 
program  variation.  Again,  this  result  is  not  as  counterintuitive  as  it 
first  appears.  Prototyping  may  result  in  identification  of  technical 


®JcfTrey  A.  Drezner  and  Giles  K.  Smith,  An  Analysis  of  Weapon  System  Acquisition 
Schedules,  RAND,  R-3937-ACQ,  December,  1990. 
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SOURCE;  Selected  Acquisition  Reports  as  of  December  1988. 
NOTE:  Slip  measured  as  deviation  in  first  operational  delivery. 


Figure  5.2— Prototyping  and  Schedule  Slip 


deficiencies  earlier  in  a  program,  thus  causing  schedule  slip  while  ac¬ 
tually  improving  overall  project  outcome.  Nonprototyping  programs 
might  not  uncover  the  same  problem  until  well  into  the  production 
phase.  There  is  currently  no  database  available  that  would  allow 
definitive  testing  of  this  notion. 


prototyping  and  cost  growth 

As  with  schedule  slip,  cost  growth  depends  on  when  the  results  of 
prototyping  are  used  in  managing  the  program.  If  a  cost  baseline  was 
established  using  the  information  generated  by  prototyping,  we  would 
expect  improved  cost  estimation  accuracy,  thus  reducing  cost  growth 
as  measured  from  the  baseline  estimate.  If  that  baseline  was  estab¬ 
lished  either  prior  to  the  prototyping  activities,  or  without  using  in¬ 
formation  available  through  prototyping  activities,  then  prototyping 
activities  that  generated  information  resulting  in  increased  cost  esti¬ 
mates  would  be  expected  to  increase  cost  growth  measured  from  that 
baseline.  Figure  5.3  shows  the  total  program  acquisition  cost  growth 
of  prototyping  and  nonprototyping  programs.^  The  distribution  again 


<Co8t  growth  is  measured  as  the  dilTerence  between  a  baseline  estimate  and  the 
most  current  estimate  or  actual  program  costs,  after  correcting  for  inflation  and 


shows  a  high  variation  in  cost  growth  outcomes  between  prototyping 
and  nonprototyping  progpi'ams  for  any  level  of  program  maturity 
(measured  as  years  past  FSD  start).  The  dollar-weighted  averag*^ 
cost-growth  ratio  for  the  protot3rping  programs  in  Figure  5.3  is  1.30, 
while  nonprototyping  programs  average  1.16,  a  fairly  large  differ¬ 
ence.® 

A  similar  rationale  suggests  that  prototyping  would  have  a  greater 
benefit  earlier  in  a  program  rather  than  later  because  information  al¬ 
lowing  improved  estimates  is  made  available  earlier.  However,  only  a 
slight  difference  was  found  between  the  cost  growth  of  programs  that 
prototyped  prior  to  FSD  (1.30)  and  after  FSD  start  (1.34). 


Years  past  start  of  full-scale  development 

SOURCE:  Raw  data  from  Selected  Acquisition  Reports  as  of  December  1 988. 

Figure  5.3 — Prototyping  and  Cost  Growth 
(Development  Estimate  Baseline) 


quantity  changes.  A  factor  greater  than  one  indicates  cost  growth.  These  data,  derived 
from  Selected  Acquisition  Reports,  are  based  on  ongoing  work  by  the  author. 

^Total  program  cost  at  the  time  of  the  development  estimate  is  used  as  the  program 
weight  in  these  calculations.  All  cost  growth  figures  are  rcferenoed  to  the  development 
estimate  baseline. 

^These  figures  include  all  programs  for  which  it  was  possible  to  classify  prototyping 
vs.  nonprototyping.  A  subset  of  this  database — programs  in  which  the  prototyping  vs. 
nonprototyping  distinction  can  be  made  with  higher  confidence — shows  similar  results, 
though  of  smaller  magnitude.  For  the  high  confidence  subset,  prototyping  programs 
had  a  dollar-weighted  average  cost-growth  factor  of  1.26,  and  nonprototyping  programs 
had  1.19. 
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We  can  also  examine  the  cost  growth  associated  with  the  research, 
development,  test  and  evaluation  (RDT&E),  or  procurement  accounts 
for  these  programs  separately.  We  might  expect  to  observe  relatively 
higher  R&D  cost  growth  and  lower  production  cost  growth  in  proto¬ 
typing  programs,  indicating  identification  and  mitigation  of  problems 
prior  to  production.  Nonprototyping  programs  should  incur  relatively 
higher  cost  growth  in  the  procurement  account,  while  prototyping 
programs  should  incur  higher  RDT&E  cost  growth.  Figure  5.4  shows 
this  comparison.  For  nonprototyping  programs,  the  dollar-weighted 
average  cost  growth  ratio  for  RDT&E  is  1.18,  while  prototyping  pro¬ 
grams  averaged  1.43,  a  significant  difference  in  the  expected  direc¬ 
tion.  The  production  phase  averages  are  1.17  and  1.29  for  nonproto¬ 
typing  and  prototyping  programs,  respectively.  These  results  do  not 
provide  evidence  to  support  the  hypothesis  that  prototyping  increases 
R&D  cost  growth  and  reduces  procurement  cost  growth.’ 
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SOURCE:  Raw  data  from  Selected  Acquisition  Reports  as  of  December  1988. 


All  programs 


Prototyping 


Nonprototyping 


Figure  5.4 — Comparison  of  RDT&E  and  Procurement  Cost  Growth 
(Prototyping  and  Nonprototyping  Programs) 


’Again,  for  the  high  confidence  subset  of  the  database,  results  are  similar,  as  the 
table  below  shows: 


RDT&E  Procurement 


All  programs 

(n 

=  52) 

1.32 

1.20 

Prototyping 

(n 

=  29) 

1.36 

1.23 

Nonprototyping 

(n 

=  23) 

1.29 

1.19 
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Table  5.2  presents  a  comparison  of  dollar-weighted  average  cost 
growth  for  prototyping  and  nonprototyping  programs  by  weapon  sys¬ 
tem  type.  Both  development  and  procurement  cost  growth  are  shown, 
as  well  as  cost  growth  for  the  total  program.  The  table  demonstrates 
considerable  variability  in  all  three  measures  of  cost  growth  across 
weapon  types.  We  need  to  use  caution  when  interpreting  the  data  in 
the  table,  however,  because  of  the  small  number  of  observations  in 
each  category.  For  instance,  while  total  program  cost  growth  for  air¬ 
craft  averages  18  percent  for  programs  that  included  prototyping 
activities  and  27  percent  for  nonprototyping  programs,  the  result  is 
entirely  dominated  by  a  single  program:  the  F-111.  The  F-111  is  a 
large  program  with  a  high  cost-growth  factor,  so  it  tends  to  dominate 

Table  5.2 


Cost  Growth  by  Weapon  Type:  Prototyping  vs.  Nonprototyping 


Development 

Procurement 

Total  Program 

Proto, 

(#  obs.) 

Nonproto. 
(#  obs.) 

Proto. 

(#  obs.) 

Nonproto. 
(#  obs.) 

Proto. 

(#  obs.) 

Nonproto. 
(#  obs.) 

All  programs 

1.43 

1.18 

1.28 

1.17 

1.30 

1.16 

(45) 

(37) 

(45) 

(37) 

(44) 

(46) 

Aircraft 

1.40 

1.31 

1.13 

1.26 

1.18 

1.27 

(5) 

(9) 

(5) 

(9) 

Missile 

1.47 

1.02 

1.31 

1.07 

1.35 

1.05 

(22) 

(8) 

(22) 

(8) 

Helicopter 

1.35 

n/a 

1.39 

n/a 

1.39 

n/a 

(4) 

(4) 

Electronic 

1.39 

1.23 

1.39 

1.12 

1.32 

1.16 

(6) 

(10) 

(6) 

(10) 

Munition 

1.28 

n/a 

1.19 

n/a 

1.20 

n/a 

(5) 

(5) 

Vehicle 

2.15 

n/a 

1.60 

n/a 

1.70 

n/a 

(2) 

(2) 

Ship 

n/a 

1.19 

n/a 

1.07 

n/a 

1.06 

(8) 

(8) 

Space 

.99 

1.34 

1.13 

1.10 

1.05 

1.22 

(1) 

(2) 

(1) 

(2) 

Other 

n/a 

n/a 

n/a 

r/a 

n/a 

n/a 

SOURCE:  Raw  data  from  Selected  Acquisition  Reports  as  of  December  1988. 
Analysis  from  ongoing  work  by  author. 

NOTE:  Data  are  dollar-weighted  averages  for  programs  three  or  more  years  past 
PSD  start  measured  from  the  development  estimate  baseline. 
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a  dollar-weighted  calculation.  Removing  the  F-111  results  in  a  13 
percent  cost-growth  average,  a  significant  reduction  and  a  change  in 
the  relative  magnitudes  of  average  cost  growth  for  aircraft  systems. 

Table  5.2  points  to  some  interesting  issues  that  need  to  be  more  fully 
explored.  For  instance,  the  database  seems  to  suggest  that  all  heli¬ 
copter  and  munition  systems  are  prototyped  while  no  ships  are. 
Though  we  might  suspect  that  ships  are  not  amenable  to  prototyping, 
it  seems  luilikely  that  there  is  something  in  the  technical  nature  of 
helicopters  and  munitions  that  would  require  prototyping.  Hence, 
this  result  should  not  be  taken  as  definitive  without  further  research. 
Similarly,  it  appears  that  significant  cost-growth  differences  between 
prototyping  and  nonprototyping  programs  appear  in  missile  and  elec¬ 
tronic  systems,  with  prototyping  programs  incurring  higher  develop¬ 
ment  and  procurement  cost  growth.  The  current  database  does  not 
support  a  detailed  explanation  of  this  apparent  phenomena. 

SUMMARY 

The  evidence  linking  prototyping  with  program  outcomes  is  fairly 
weak.  In  most  cases,  cost  and  schedule  outcomes  vary  widely,  with  no 
demonstrated  relationship  between  prototyping  and  outcome.  Most  of 
the  differences  between  prototyping  and  nonprototyping  programs 
were  not  significant  from  a  statistical  standpoint.  It  might  be,  for  in¬ 
stance,  that  only  the  more  technically  challenging  programs  include 
prototyping  activities,  in  which  case  we  might  expect  higher  risk  even 
after  prototyping.  In  short,  we  do  not  know  enough  about  the  factors 
affecting  program  outcomes  to  know  whether  prototyping  has  an  ef¬ 
fect.  The  likelihood  is  that  many  other  factors  dominate  any  effect  of 
prototyping  on  program  outcomes.  Such  factors  as  budget  stability, 
technical  difficulty,  and  perhaps  the  ability  and  willingness  of  deci¬ 
sion  makers  and  program  managers  to  use  improved  information  in 
cost  and  schedule  estimates  are  likely  to  dominate  any  comparison  of 
prototyping  and  nonprototyping  programs. 

An  interesting  hypothesis  that  deserves  attention  in  future  research 
is  the  notion  that  choosing  an  appropriate  prototyping  strategy  may 
make  the  difference  between  positive  and  negative  outcomes  for  a 
given  program.  Unfortunately,  this  again  raises  the  problem  de¬ 
scribed  at  the  beginning  of  this  section:  We  cannot  know  what  the 
outcome  of  a  prototyping  program  would  have  been  if  it  was  not  pro¬ 
totyped;  we  can  only  speculate.  Further,  the  currently  available  data 
do  not  support  an  analysis  of  the  appropriatenesf  of  a  particular  pro¬ 
totyping  strategy  for  a  particular  program. 


6.  OBSERVATIONS  ON  PROTOTYPING  POLICY 


SUMMARY  OF  FINDINGS 

The  basic  results  of  this  analysis  of  the  nature  and  role  of  prototyping 
in  weapon  system  development  can  be  summarized  as  follows: 

•  Prototyping  is  a  complex  family  of  strategies,  with  evidence  of  wide 
variation  in  past  use.  There  is  no  single  generic  approach  to  proto¬ 
typing  that  is  universally  applicable. 

•  A  few  prototyping  strategies  (combinations  of  three  basic  elements: 
goals,  timing,  and  level  of  integration)  appear  to  be  most  common. 

•  There  are  few  identifiable  differences  in  prototyping  strategy  be¬ 
tween  services  or  across  weapon  types. 

•  Detailed  programmatic  characteristics,  such  as  contract  type,  re¬ 
quirements  specification,  number  of  prototypes,  decision  layers, 
and  amount  of  documentation,  are  loosely  associated  with  the  basic 
strategy  elements  and  tend  to  flow  from  the  choice  of  those  ele¬ 
ments. 

•  Macro-level  strategies  have  been  fairly  constant  over  time,  though 
some  factors  may  have  changed  at  the  detailed  level. 

•  The  effect  of  prototyping  on  program  cost,  schedule,  and  perfor¬ 
mance  outcomes  is  ambiguous  due  to  the  effect  of  confounding 
variables. 

This  section  goes  beyond  these  results  and  begins  to  address  the  more 
normative  issue  of  what  the  characteristics  of  prototyping  policy 
should  be.  While  it  draws  on,  and  is  informed  by,  these  findings,  it 
adds  other  arguments  that  are  not  necessarily  demonstrated  by  the 
analysis  presented  earlier.  Hence,  this  section  should  be  viewed  as  a 
more  subjective  and  qualitative  treatment  of  prototyping  issues, 
somewhat  divorced  from  the  analysis  just  presented.  It  addresses  the 
elusive  normative  goal  of  formulating  useful  prototyping  policy. 

PROBLEMS  AND  PROSPECTS  FOR  FIRM  CRITERIA 

Policy  makers  have  often  sought  criteria  to  help  set  effective  and  un¬ 
ambiguous  policy  on  when  prototyping  should  be  used  in  the  course  of 
weapon  system  development  and  what  kind  of  activities  prototyping 
should  entail.  Though  many  attempts  have  been  made  to  develop 
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such  criteria,  they  have  met  with  little  success.  The  research  just 
presented  suggests  one  possible  reason:  the  lack  of  an  unambiguous, 
agreed-on  definition  of  what  a  prototype  is.  In  other  words,  lack  of  a 
generally  accepted  definition  of  prototjrping,  including  potential  bene¬ 
fits,  makes  consistent  policy  formulation  and  implementation  diffi¬ 
cult.  Thus,  there  remains  a  lack  of  consistent  guidance  for  policy  and 
practice  on  the  use  of  prototyping. 

We  offer  a  broad  definition: 

A  prototype  is  a  tangible  product  (hardware  and/or  software)  that  al¬ 
lows  hands-on  testing  in  a  realistic  environment.  In  scope  and  scale,  it 
represents  a  concept,  subsystem,  or  production  article  with  potential 
utility.  It  is  not  necessarily  a  complete  system,  but  rather  focuses  on 
those  high-risk  areas  critical  to  system  success. 

The  key  for  policy  guidance,  however,  is  that  a  prototype  is  intended 
to  generate  information  to  improve  the  quality  of  decisions  in  an  envi¬ 
ronment  of  risk  and  uncertainty,  not  merely  to  demonstrate  satisfac¬ 
tion  of  contractual  performance  specifications.  Hence,  a  successful 
prototype  program  may  lead  to  cancellation  of  a  proposed  system,  a 
reduction  in  its  level  of  technological  advance,  or  continuation  of  de¬ 
velopment  as  planned. 

Though  we  have  intentionally  defined  prototypes  in  the  broadest  pos¬ 
sible  manner,  one  important  aspect  needs  clarification:  the  timing  of 
the  prototyping  phase  within  the  acquisition  process.  We  argue  that 
prototyping  can,  and  does,  occur  both  before  and  after  Milestone  II. 
Full-scale  (but  not  necessarily  full  system)  prototyping  usually  occurs 
before  the  start  of  engineering  and  manufacturing  development 
(EMD),  as  in  the  classic  case  of  the  Light  Weight  Fighter  program 
(YF-16/17)  or  the  recently  completed  ATF  demonstration  validation 
phase.  Subsystem  level  prototyping,  while  not  “full  scale”  in  the  same 
sense,  can  and  does  occur  both  before  and  after  beginning  EMD.  That 
distinction  is  important  to  our  basic  policy  recommendations. 

Our  analysis  of  past  prototyping  experience  has  documented  the  wide 
range  of  uses  prototypes  have  in  weapon  system  development.  That 
extreme  variation  in  prototyping  strategies  is  expressed  at  the  high¬ 
est  level  in  terms  of  the  goals  of  the  prototyping  activities,  the  timing 
of  those  activities  with  respect  to  the  overall  program  schedule,  and 
the  level  of  system  integration  of  the  prototype(s).  While  there  are 
different  ways  to  use  prototyping,  all  are  intended  to  reduce  risk  and 
uncertainty  of  one  sort  or  another.  Our  systematic  analysis  of  proto¬ 
typing  confirms  what  many  in  the  acquisition  field  have  learned  case 
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by  case:  that  risk  and  uncertainty — and  thus  information  needs — 
vary  dramatically  from  program  to  program.  That  variation  hinders 
development  of  specific  prototyping  criteria  that  can  be  universally 
applied. 

Theoretically,  when  deciding  whether  or  not  to  protot3T)e,  one  would 
compare  the  costs  of  prototyping  in  a  particular  program  with  its  ben¬ 
efits.  Figure  6.1  illustrates  this  deceptively  simple  concept.  We  will 
only  want  to  prototype  when  the  benefits  of  prototyping  are  greater 
than  or  equal  to  the  costs.  Hence,  for  any  point  above  the  45  degree 
line  in  Figure  6.1,  we  would  prototype;  for  any  point  below  it,  we 
would  not.  Unfortunately,  this  conceptual  decision  framework  is  not 
simple  to  put  in  practice.  In  particular,  both  costs  and  benefits  are 
multidimensional. 

There  are  several  ways  in  which  costs  can  (and  should)  be  considered. 
The  obvious  is  in  terms  of  dollars,  both  the  actual  expenditure  in¬ 
volved  in  the  prototyping  activities  and  as  the  percentage  of  total 
program  acquisition  costs.  But  there  are  additional  costs  to  be  con¬ 
sidered.  Time  is  a  form  of  cost,  for  example,  though  it  may  not  be 
measured  in  dollars.  There  are  political  costs  as  well.  For  instance, 
taking  the  time  to  demonstrate  technology  might  give  opponents  of  a 
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program  time  to  gather  support  to  terminate  or  restructure  it.  Simi¬ 
larly,  a  prototyping  activity  that  demonstrates  that  a  technology  is 
not  mature  enough  to  be  incorporated  in  a  new  system  might  also 
provide  a  basis  for  increased  political  opposition  to  the  entire  project, 
even  if  the  relative  level  of  technological  advance  of  the  system  was 
reduced  as  a  result  of  information  gained  through  prototyping. 

Benefits  are  perhaps  even  more  difficult.  For  instance,  benefits  may 
be  thought  of  as  a  reduction  in  development  risks.  But  we  cannot 
quantify  how  much  risk  is  reduced.  Nor  can  we  know  exactly  what 
kinds  of  risks  are  reduced  for  each  type  of  prototyping  activity.  Risks 
might  be  stated  as  the  cost  consequences  of  proceeding  into  the  next 
progrram  phase  with  a  poor  design.  That  has  two  components:  the 
costs  of  an  undesired  event  happening  because  of  the  poor  design,  and 
the  probability  that  the  event  will  occur.  Both  components  are  highly 
uncertain.  There  are  other  benefits  to  prototyping  as  well,  such  as 
the  possibility  that  more  accurate  cost,  schedule,  and  performance  es¬ 
timates  will  allow  better-quality  decisions  regarding  cost,  schedule, 
and  performance  trade-offs,  and  increase  design  options. 

In  summary,  there  are  tremendous  difficulties  involved  in  opera¬ 
tionalizing  the  simple  concept  depicted  in  Figure  6.1.  We  cannot  be 
confident  that  we  can  identify  all  possible  benefits  and  costs.  Each 
benefit  and  cost  is  measured  on  a  different  scale,  if  it  can  be  measured 
at  all,  and  we  do  not  know  how  to  reconcile  those  scales  consistently 
across  all  the  dimensions  of  costs  and  benefits.  Even  if  we  could  mea¬ 
sure  all  relevant  costs  and  benefits,  we  do  not  know  how  to  consis¬ 
tently  weigh  them  in  a  decision  process.  Differences  in  program 
characteristics  (e.g.,  technological  difficulty  and  maturity,  cost  and 
schedule  constraints,  the  level  of  uncertainty  regarding  the  technolo¬ 
gy’s  military  utility,  etc.)  suggest  that  those  weights  will  differ  greatly 
across  programs  and  also  as  a  function  of  the  goals  and  activities  in¬ 
volved  in  a  particular  prototyping  application. 

TOWARD  MORE  FLEXIBLE  POLICY  GUIDANCE 

Nonetheless,  several  general  principles  seem  valid  in  prototyping  pol¬ 
icy.  The  basic  policy  guideline  might  be  to  assess  the  extent  to  which 
prototyping  is  both  useful  and  appropriate  in  a  particular  case.  One 
fundamental  consideration  is  the  trade-off  between  the  net  benefits 
(benefits  minus  costs  across  all  relevant  dimensions)  of  prototyping 
and  the  type  and  magnitude  of  risk  in  a  development  effort,  as  illus¬ 
trated  in  the  discussion  of  Figure  6.1.  Table  6.1  lists  just  a  few  of  the 
factors  that  need  to  be  carefully  evaluated  in  assessing  that  trade-off; 
it  is  certainly  not  meant  to  be  exhaustive.  Hence,  prototyping  policy 
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Table  6.1 

Factors  to  Consider  in  Making  Prototyping  Decisions 

Benefits  from  Prototyping 

•  Reduces  technological  risk  and  uncertainty 

*  Identifies  critical  system  integration  issues 

■  Permits  more  accurate  cost,  schedule,  and  performance  estimates 

*  Reduces  cost  consequences  of  proceeding  into  subsequent  phases  with  poor  design 

•  Allows  neccssaiy  design  changes  to  be  identified  early 

Costs  of  Prototyping 

■  Increased  front  end  cost 

*  Prototyping  phase  cost  as  percent  of  total  program  cost 

•  Longer  front-end  time  (interval  for  prototyping  phase) 

might  require  the  evaluation  and  balancing  of  the  relative  technologi¬ 
cal  advance  in  a  system,  uncertainty  regarding  the  utility  of  a  new 
concept,  and  the  maturity  of  the  technology  being  incorporated,  given 
cost  and  schedule  constraints.  Unfortunately,  there  is  no  way  to  con¬ 
sistently  measure  those  dimensions.  Further,  even  if  we  could  accu¬ 
rately  measure  the  relative  technological  maturity  of  a  system  design, 
for  instance,  we  still  do  not  know  how  much  weight  to  give  any  single 
factor  in  the  overall  decision  calculus.  The  policy  guidance  we  are 
suggesting  is  that  these  and  other  similar  factors  be  explicitly  in¬ 
cluded  in  the  decision  process  when  a  program’s  acquisition  strategy 
is  being  formulated.  We  are  not  specifying  exactly  which  factors  are 
important  or  how  they  should  enter  the  decision  process.  In  the  end, 
there  is  no  substitute  for  informed  judgment  made  by  experienced 
managers  and  engineers. 

The  decision  process  we  have  in  mind  here  can  be  illustrated  (rather 
simplistically)  with  a  brief  example.  The  Advanced  Medium  Range 
Air-to-Air  Missile  (AMRAAM)  program  presents  a  classic  case  of 
when  prototyping  should  have  been  done  and  in  fact  was  done.*  AM¬ 
RAAM  incorporated  significant  technological  advance,  with  the  asso¬ 
ciated  risk;  it  was  a  large  program  in  terms  of  both  quantity  and  cost, 
and  full-scale  prototyping  could  be  performed  in  a  competitive  envi¬ 
ronment  with  articles  built  on  soft  tooling.  In  contrast,  the  B-2  pro¬ 
gram  is  a  case  in  which  the  large  unit  cost,  small  production  quantity, 
and  technical  challenge  and  cost  of  fabricating  even  development 
items  (full  production  tooling  is  required)  makes  full-scale  prototyping 

*This  evaluation  is  independent  of  the  observed  cost  and  schedule  growth  in  the 
program,  which  may  in  part  be  attributed  to  failure  to  completely  incorporate  the 
information  generated  by  the  prototyping  phase  into  structuring  later  phases. 
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unattractive.  Nonetheless,  subsystem  and  component-level  prototyp¬ 
ing  could  still  be  done,  and  was. 

How  can  we  determine  the  dividing  line  between  situations  that  do 
and  do  not  justify  prototyping?  Our  research  on  prototyping  has  not 
produced  unambiguous  analytical  results.  However,  during  the 
course  of  our  research  on  this  and  related  topics,  we  have  concluded 
that  some  form  of  prototyping  is  almost  always  appropriate.  This  is 
basically  because  there  are  powerful  institutional  pressures  that  lead 
to  systematic  underestimation  of  program  risks.  We  believe  that  the 
policy  should  be  to  use  some  form  of  prototyping  in  almost  every  case, 
and  the  burden  of  proof  should  be  on  those  who  argue  that  a  prototyp¬ 
ing  activity  is  unnecessary  or  impractical.  In  cases  where  full-scale 
articles  are  too  expensive  or  technically  impractical,  subsystem  proto- 
typing  (e.g.,  avionics  test  beds)  might  still  be  appropriate.  While  less 
than  full-scale  prototyping  may  not  capture  the  complete  set  of  ben¬ 
efits  attributable  to  prototyping  (e.g.,  system  integration  risk  cannot 
be  adequately  addressed),  it  still  offers  a  net  benefit  to  the  program. 

This  conceptual  framework  for  policy  and  decision  making  is  quite 
similar  to  what  is  contained  in  existing  regulations.  DoD  Instruction 
5000.2  mandates  competitive  prototyping  for  all  major  weapon  sys¬ 
tems,  in  accordance  with  law.*  The  regulation  states  that  require¬ 
ments  for  prototyping  (e.g.,  a  prototyping  strategy  in  our  terminology 
here)  will  be  established  at  the  Milestone  I  decision  point  based  on  a 
program-specific  risk  assessment.  The  regulation  even  allows  for  pro¬ 
totyping  activities  during  FSD/EMD,  a  contentious  point  in  the  past. 
Further,  the  Under  Secretary  of  Defense  for  Acquisition,  Donald 
Yockey,  has  delineated  criteria  in  a  fashion  paralleling  that  presented 
here: 

We  require  the  use  of  competitive  prototyping  in  every  case  in  which  it 
makes  sound  business  sense.  .  .  .  Our  policy  is  to  require  competitive 
prototyping  whenever  the  risk  avoidance  to  be  gained  outweighs  the 
costa.^  (Italics  added.) 

The  underlying  law  states  that  the  requirement  for  competitive  pro¬ 
totyping  may  be  waived  by  the  Secretary  of  Defense  based  on  cost 
considerations.*  Though  the  regulation  applies  to  only  one  form  of 
prototyping  (competitive),  the  criteria  is  necessarily  general  enough  to 


^Scc  DoDI  5000.2,  February  1991,  p.  5-D-2  and  Title  10,  United  States  Code, 
Section  2365,  paragraph  (a). 

^Inside  the  Pentagon,  June  13, 1991,  p.  12. 

^Title  10,  United  States  Code,  Section  2365,  paragraph  (c). 
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apply  to  other  forms  of  prototyping  activities  as  well.  From  the 
perspective  of  this  research,  the  expressed  policy  of  DoD  is  consistent 
with  the  conceptual  framework  developed  both  in  this  section  and  in 
Section  2. 

The  basic  implication,  then,  is  that  acquisition  policy  should  reflect 
only  broad  guidelines  on  prototyping,  rather  than  specifying  detailed 
strategies  or  criteria.  This  statement  is  not  based  on  empirical  evi¬ 
dence,  since  we  could  not  effectively  relate  prototyping  to  program 
outcomes.  Rather,  it  follows  from  the  nature  and  purpose  of  prototyp¬ 
ing — which  is  highly  variable — ^that  there  would  be  no  blanket  policy 
that  could  cover  all  applications  of  prototyping.  There  is  no  single  ap¬ 
proach  to  prototyping;  effective  practice  involves  considerable  flexibil¬ 
ity,  both  in  tailoring  a  particular  strategy  to  the  needs  of  a  develop¬ 
ment  effort  and  in  using  the  resulting  information. 

We  recommend  that  prototyping  be  explicitly  considered  as  part  of 
the  strategy  for  development  of  a  weapon  system,  but  that  acquisition 
policy  should  reflect  only  broad  guidelines  on  prototyping,  rather  than 
attempting  to  specify  detailed  prototyping  strategies.  In  other  words, 
we  advocate  including  the  full  range  of  prototyping  considerations  in 
a  rational  decision  process  along  the  lines  described  here  and  illus¬ 
trated  in  Figure  6.1  and  Table  6.1.  There  might  be  situations  in 
which  prototyping  is  not  appropriate.  Though  such  cases  may  be 
rare,  when  they  do  occur,  the  decision  process  we  are  suggesting  will 
allow  explicit  rationalization  as  to  why  prototyping  was  deemed  inap¬ 
propriate. 

There  may  be  other  reasons  to  prototype  as  well.  For  instance,  cer¬ 
tain  forms  of  prototyping  can  create  and  preserve  technological  and 
system  options,  maintain  and  enhance  both  the  technology  and  indus¬ 
trial  base  important  to  the  quality  of  defense  products,  and  hedge 
against  greater  uncertainty  in  threats.  For  instance,  the  interaction 
of  three  recent  trends  suggests  a  possible  new  utility  for  prototyping: 
limited  and  declining  budgets,  an  uncertain  and  rapidly  changing  mil¬ 
itary  and  political  environment,  and  the  need  to  sustain  a  viable 
technology  and  industrial  base.  Prototyping  is  a  strategy  that  can 
theoretically  address  these  concerns.  It  is  generally  less  expensive 
than  a  full-scale  development,  it  requires  a  supporting  technology  and 
production  base,  and  it  provides  crucial  experience  to  design  engi¬ 
neers  and  managers  in  both  industry  and  government.  Though  im¬ 
portant,  such  factors  are  broader  in  scope  than  the  issues  discussed 
above,  which  focus  on  the  application  of  prototyping  to  specific 
weapon  system  developments.  Nevertheless,  future  research  should 
consider  these  issues  in  more  detail. 
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Additionally,  both  the  DoD  and  Congress  should  encourage  a  proper 
environment  for  prototyping.  If  prototyping  is  to  be  a  useful  strategy, 
the  information  generated  must  be  incorporated  into  decisions.  This 
implies  the  need  for  a  great  deal  of  flexibility  by  program  managers  in 
both  government  and  industry,  and  a  willingness  to  use  the  results  of 
prototyping  activities  to  make  cost,  schedule,  and  performance  trade¬ 
offs.  Unfortunately,  many  officials  (in  the  Office  of  the  Secretary  of 
Defense  (OSD),  the  services,  and  Congress)  do  not  understand  that  a 
prototype  is  successful  when  it  identifies  problems.  By  its  nature, 
prototyping,  properly  applied,  will  produce  some  “failures.”  It  needs  to 
be  recognized  that  prototyping  is  oriented  toward  identifying  prob¬ 
lems  prior  to  full  commitment  to  the  program. 


Appendix  A 

PROGRAMS  INCLUDED  IN  THE  DATABASE 


The  following  list  identifies  the  programs  used  in  this  analysis.  Aside 
from  the  program  name,  some  additional  information  is  provided. 
The  year  ■'f  first  operation  identifies  the  year  in  which  the  prototype 
was  (or  will  be)  first  operated.  Using  the  narrower  definitional  crite¬ 
ria  discussed  in  Section  2,  an  assessment  was  made  as  to  whether  a 
program  meets  this  prototype  definition  or  not,  and  whether  there 
was  enough  information  available  to  make  this  determination.  This 
addresses  the  problem  of  consistent  definitions:  The  use  of  the  term 
prototype  varies  considerably  in  the  literature.  Lastly,  a  subjective 
assessment  of  the  data  quality  and  sources  is  made  with  respect  to 
understanding  the  role  and  rationale  of  the  prototype  in  the  program. 

Table  A.l  lists  the  programs  included  in  the  literature  database. 
Programs  were  identified  for  potential  inclusion  if  one  or  more 
sources  indicated  that  they  had,  or  were  planning  to  have,  some  type 
of  prototyping  activity  during  the  course  of  development.  Most  of 
these  programs  were  included  in  the  database  after  assessing  the  rea¬ 
sonableness  of  the  prototyping  designation  against  the  definition  used 
in  this  study.  The  only  other  criterion  used  to  determine  whether  or 
not  to  include  a  program  was  that  the  prototype  was  first  operated  af¬ 
ter  1960.  Though  the  result  is  not  a  complete  list,  it  is  considered 
representative  of  prototyping  activities  in  the  post- 1960  period. 

Table  A.2  lists  the  programs  included  in  the  program  manager  sur¬ 
vey.  The  prototyping  designations  for  these  programs  are  consistent 
with  those  of  the  literature  database  if  the  system  referenced  in  the 
survey  is  the  same  as  that  identified  in  the  literature.  Worksheet 
questionnaires  were  sent  to  85  U.S.  government  program  managers: 
43  responses  were  received.*  In  general,  the  quality  of  the  data  pro¬ 
vided  by  the  survey  is  superior  to  that  of  the  public  literature.  While 
the  weapon  system  classifications  that  are  indicated  can  be  argued 
(e.g.,  the  V-22  as  an  aircraft  rather  than  a  helicopter),  the  classifica¬ 
tions  are  those  that  were  indicated  by  the  respondent. 


*GPS  returned  three  separate  worksheets,  one  each  for  the  space,  user  equipment, 
and  ground  control  segments  of  the  program.  These  are  treated  separately  here  as 
they  each  had  distinct  acquisition  (and  prototyping)  strategies. 
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Table  A.1 

Programs  in  the  Literature  Review  Database 


Program  Name 

Year  First 
Operation 

Definition^ 

Information 

Aircraft; 

X-29A 

1984 

Yes 

Good 

X-30  National  Aerospace  Plane 

1993 

Yes 

Adequate 

X-31A 

1989 

Yes 

Adequate 

Advanced  Fighter  Integration  Program 

1976 

Yes 

Adequate 

Agile  Falcon 

No 

Poor 

YF-12/A-11 

1964 

Yes 

Poor 

SR-71A/B 

1964 

7 

Poor 

YF-7F  (A-7  Plus) 

1988 

Yes 

Adequate 

A-6F 

1987 

No 

Adequate 

F-20 

1982 

Yes 

Good 

Skytradcr  Scout 

7 

Poor 

Hypersonic  Glide  Vehicle 

Yes 

Poor 

AMX 

1984 

Yes 

Good 

Lavi 

1986 

Yes 

Good 

Experimental  Aircraft  Program 

1986 

Yes 

Good 

European  Fighter  Aircraft 

1991 

Yes 

Good 

Rafale  A 

1986 

Yes 

Good 

Rafale  D 

1990 

Yes 

Good 

Hawk  200 

1986 

7 

Poor 

Gripen  JAS-39 

1987 

No 

Adequate 

AC- 130 

Yes 

Poor 

B-70 

1964 

Yes 

Good 

C-141A 

1963 

7 

Adequate 

C-141B(YC.141B) 

1977 

Yes 

Poor 

OV-lOA 

1965 

Yes 

Good 

Charger 

1964 

Yes 

Good 

T-46A 

1985 

No 

Adequate 

KC-lOA 

7 

Poor 

KC.135R 

1982 

7 

Poor 

A- 10  (YA-WlO) 

1972 

Yes 

Good 

r  3A 

1972 

Yes 

Good 

B-IA 

1974 

Yes 

Adequate 

B-IB 

1984 

No 

Adequate 

E-2C 

1971 

7 

Poor 

S-3A 

1972 

7 

Poor 

S-3B 

1984 

7 

Poor 

E-6A 

1987 

7 

Poor 

EF-111 

1977 

7 

Poor 

F-14 

1970 

No 

Adequate 

F-15 

1972 

7 

Poor 

F-15E 

1986 

Yes 

Adequate 

F-15  STOL 

1988 

Yes 

Adequate 

F-16  (YF- 16/17) 

1974 

Yes 

Good 

F-16XL(F-16E)  SCAMP 

1982 

Yes 

Adequate 

F-18L 

7 

Poor 

79 


i 


Table  A.1  (continued) 


Program  Name 

Year  First 
Operation 

Definition® 

Information 

F/A-18 

1978 

No 

Adequate 

T^STS 

? 

Poor 

Advanced  Technology  Bomber 

1988 

? 

Poor 

F-19 

? 

Poor 

Advanced  Tactical  Fighter 

1990 

Yes 

Good 

Advanced  Tactical  Aircraft 

1990 

7 

Poor 

C-17 

1988 

No 

Poor 

Advanced  Technology  Tactical  Transport 

1988 

Yes 

Good 

Bromon  BR-2000 

1988 

Yes 

Good 

AMST(YC-14/15) 

Yes 

Poor 

P-3C 

1969 

No 

Poor 

MB.339A 

1976 

7 

Poor 

ATM  42 

1984 

7 

Poor 

Nimrod 

1967 

7 

Poor 

C.lOl 

1977 

7 

Poor 

C-212 

1971 

Yes 

Adequate 

C-235 

1983 

Yes 

Adequate 

ATL.2 

1961 

7 

Poor 

Mirage  2000 

1978 

7 

Poor 

Mirage  4000 

1979 

7 

Poor 

Super  Etcndard 

1974 

7 

Poor 

EMB-312 

1980 

7 

Poor 

FMA  lA  58  Pucara 

1969 

7 

Poor 

FMA  LA  63  Pampa 

1984 

7 

Poor 

Arhens 

7 

Poor 

HAL  Light  Combat  Aircraft 

7 

Poor 

Seastar 

1984 

7 

Poor 

A-4S-1 

7 

Poor 

A4K 

1987 

7 

Poor 

F1300  Jet  Squalus 

7 

Poor 

V-22 

1988 

7 

Adequate 

XV-15 

Yes 

Adequate 

V/STOL 

7 

Poor 

XC-142 

1963 

Yes 

Adequate 

X-100 

1960 

Yes 

Adequate 

X-19 

1964 

Yes 

Adequate 

X-22A 

1966 

Yes 

Adequate 

CLr84 

1965 

Yes 

Adequate 

CL^-1 

1970 

Yes 

Adequate 

SC.l 

1960 

Yes 

Adequate 

D0.31E 

1967 

Yes 

Adequate 

XV-4B 

1968 

Yes 

Adequate 

VAK191B 

1971 

Yes 

Adequate 

XV-6A 

1965 

Yes 

Adequate 

XV-4A 

1962 

Yes 

Adequate 

XFV.12A 

1977 

Yes 

Adequate 

PI  127 

1961 

Yes 

Good 

XV^A 

1964 

Yes 

Good 

I 
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Table  A.1  (continued) 


Program  Name 

Year  First 
Operation 

Definition^ 

Information 

Harrier 

1966 

? 

Poor 

AV-8A 

1970 

Yes 

Poor 

AV-8B 

1978 

Yes 

Poor 

Helicopters: 

CH-53E 

1974 

Yes 

Poor 

CH-54  Tarhe 

1964 

Yes 

Adequate 

CH-47D 

1979 

? 

Poor 

Advanced  Scout  Helicopter 

7 

Poor 

Advanced  Helicopter  Improvement  Program 

1984 

? 

Adequate 

XCH-62 

Yes 

Adequate 

UTTAS  (YAH-60/61) 

1974 

Yes 

Good 

AH-1  Cobra 

1965 

Yes 

Adequate 

AH-56A  Cheyenne 

7 

Poor 

AH-64  Apache  (YAH-63/64) 

1975 

Yes 

Good 

OH-6  Light  Observation  Helicopter 

1963 

Yes 

Adequate 

OH-58A  Kiowa 

1966 

Yes 

Adequate 

SH-60B  LAMPS 

1979 

7 

Poor 

VH-60 

7 

Poor 

Light  Helicopter  Experimental 

Yes 

Adequate 

NOTAR  (No  Tail  Rotor)  Helicopter 

Yes 

Adequate 

X-wing 

Yes 

Adequate 

PAH-2 

7 

Poor 

Model  360 

1987 

Yes 

Adequate 

H-76 

1985 

7 

Adequate 

AH-64B  Advanced  Apache 

1989 

7 

Poor 

A129  Mongoose 

1984 

7 

Poor 

SA  365M  Panther 

1984 

7 

Poor 

EH-101 

1987 

No 

Poor 

SA.341  Gazelle 

1967 

7 

Poor 

AS.332 

1978 

7 

Poor 

NH-90 

7 

Poor 

XH-59A/B  Advancing  Blade  Concept 

1975 

Yes 

Adequate 

C-122S 

1986 

7 

Poor 

Fuellcss  RPV 

1987 

Yes 

Adequate 

Brave  3000 

1983 

7 

Poor 

Sentinel  5000 

7 

Poor 

Missiles: 

Setter 

1984 

Yes 

Good 

Avenger 

Yes 

Adequate 

ADATS 

1983 

Yes 

Good 

Liberty 

Yes 

Adequate 

Paladin 

Yes 

Adequate 

Tracked  Rapier 

Yes 

Adequate 

Tacit  Rainbow 

7 

Adequate 

Flexible  Lightweight  Agile  Guided  Experiment 

1986 

Yes 

Adequate 

Titan  HI-C 

7 

Poor 
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Table  A.1  (continued) 


Program  Name 

Year  First 
Operation 

Definition® 

Information 

Agenda-D 

1962 

Yes 

Adequate 

Laser-Guided  Bomb 

1966 

Yes 

Good 

Mk-48 

? 

Poor 

Mk-50 

9 

Poor 

Harpoon 

1972 

? 

Poor 

Stinger 

1976 

? 

Poor 

Dragon 

1965 

9 

Poor 

Trident  11  D-5 

1987 

? 

Poor 

GLCM 

? 

Poor 

ALCM 

? 

Poor 

Tomahawk 

? 

Poor 

Sidewinder  AIM-9L 

1972 

9 

Adequate 

Sidewinder  AIM-9M 

1978 

9 

Poor 

Hellfire 

1978 

? 

Adequate 

Peacekeeper 

1983 

No 

Poor 

HARM 

9 

Poor 

AMRAAM 

1980 

Yes 

Good 

AAAM 

Yes 

Adequate 

Sea  Lance 

9 

Poor 

TOW 

1968 

? 

Poor 

Hypervelocity  Missile 

1987 

? 

Adequate 

ASAT  Space  Defense 

? 

Poor 

Advanced  Anti-Tank  Weapon  System — ^Medium 

Yes 

Adequate 

Advanced  Strategic  Missile  System 

9 

Poor 

Earth-Penetrating  Warheads 

9 

Poor 

Ballistic  Missile  Defense  System  Technology 

1979 

? 

Poor 

FOG-M 

9 

Poor 

Long-Range  Aircraft.  Intercept  Experiment 

1984 

? 

Adequate 

Rolling  Airframe  Missile 

1979 

No 

Poor 

Copperhead 

Yes 

Adequate 

Pershing  I 

9 

Poor 

Titan  I 

1960 

o 

Adequate 

HOT 

1971 

? 

Poor 

Short-Range  Anti-Tank  Weapon 

9 

Poor 

Condor 

1971 

? 

Poor 

MLRS  Multiple  Launch  Rocket  System 

1979 

Yes 

Good 

Land  Vehicles: 

M44  Howitzer 

? 

Poor 

LAV  Air  Defense 

7 

Poor 

M109 

1987 

Yes 

Good 

FGH-155 

7 

Poor 

Autonomous  Land  Vehicle 

1987 

Yes 

Good 

Advanced  Ground  Vehicle  Technology 

Yes 

Good 

Integrated  Smart  Artillery 

1986 

Yes 

Good 

Human  Factor  Howitzer  Test  Bed 

1986 

Yes 

Good 

HMMWV  M998 

1982 

Yes 

Good 
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Table  A.1  (continued) 


Program  Name 

Dagger 

XM800  ARSV 

IFV  M2/M3  Bradley 

MBT-70 

M-1 

M60A4 

DIVAD 

Leopard  I 

NFV-1  IFV 

SP122 

Heavy  Assault  Bridge 
Counter-Obstacle  Vehicle 
Improved  Vulcan 

Teleopcrated  Mobile  Anti-Armor  Platform 

Voice-Controlled  Robotic  Vehicle 

M88  Tank  Recovery 

Mobile  Electronic  Warfare  System 

Stingray  Light  Tank 

VCC-80 

Leclerc 

EE-T4  Ogum 

MCV-80 

Puma  IFV 

AMX40  MBT 

AMX30MBT 

AMX32  MBT 

EE-T2  MBT 

EE-Tl  MBT 

Aijun  MBT 

KIMBT 

ChicRan  900  MBT 
Tk-XMBT 
Mk7MBT 
C-IMBT 

76/62  Anti-Aircraft  Gun 
Fire  Ant 

TH-400  Tank  Destroyer 

SICBM  Launcher 

Mcrkava  I 

HET-70 

M-47 

6638  AVH 
AS-90  Ar.illcry 

Combat  Vehicle  Reconnaissance  (Tracked) 

Subsystems  and  Other: 

Navstar  Global  Positioning  System 
LANTIRN 


Year  First 
Operation 


Definition®  Information 


7 

Poor 

? 

Poor 

1974 

Yes 

Poor 

1967 

Yes 

Poor 

1976 

Yes 

Good 

7 

Poor 

1980 

Yes 

Good 

1961 

7 

Poor 

7 

Poor 

1984 

7 

Poor 

1984 

7 

Poor 

7 

Poor 

7 

Poor 

Yes 

Adequate 

Yes 

Poor 

7 

Poor 

7 

Poor 

1984 

Yes 

Adequate 

1987 

7 

Poor 

1987 

7 

Adequate 

1986 

7 

Poor 

1981 

7 

Poor 

1986 

7 

Poor 

1983 

Yes 

Good 

1960 

7 

Poor 

1979 

Yes 

Adequate 

1983 

7 

Poor 

1984 

7 

Poor 

1985 

7 

Poor 

1984 

7 

Poor 

1982 

Yes 

Adequate 

1985 

7 

Poor 

1985 

7 

Poor 

1987 

7 

Poor 

7 

Poor 

1985 

7 

Adequate 

1984 

7 

Poor 

7 

Poor 

1974 

Yes 

Good 

1987 

7 

Poor 

7 

Poor 

1985 

7 

Poor 

1986 

7 

Poor 

1969 

7 

Adequate 

1974 

Yes 

Good 

1983 

No 

Good 
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Table  A.l  (continued) 


Year  First 


Program  Name 

Operation 

Definition® 

Information 

Airborne  Self-Protecting  Jammer 

1983 

? 

Good 

Joint  Tactical  Information  Distribution 

1977 

System 

? 

Good 

T800  Engine 

Yes 

Good 

Joint-Technology  Demonstrator  Engine 

1989 

7 

Poor 

F404-F2J3 

7 

Poor 

ATF  Engine  (YF-119/120) 

1988 

Yes 

Good 

EJ200  Engine 

Poor 

XG-40  Engine 

1986 

7 

Poor 

XG-15  Engine 

7 

Poor 

GT601  Engine 

7 

Poor 

Advanced-Technology  Demonstrator  Engine 

7 

Poor 

Modem-Technology  Demonstrator  Engine 

7 

Poor 

Integrated  High-Performance  Engine 

Technologies  Initiative 

Yes 

Adequate 

UDF  Engine 

1988 

Yes 

Poor 

Infrared  Search  and  Track 

7 

Poor 

Common  Integrated  Processing 

1988 

7 

Adequate 

Submarine 

Yes 

Poor 

MIMIC 

7 

Poor 

Superconductors 

1987 

7 

Poor 

Thunderbolt 

1987 

Yes 

Adequate 

Caseless  Ammunition  Rifle 

1984 

Yes 

Adequate 

Close  Assault  Weapon 

7 

Adequate 

SHORAD  C2 

7 

Poor 

TADS/PNVS 

1979 

Yes 

Adequate 

SINCGARS 

7 

Poor 

TACT  AS 

Poor 

BQQ-5  Sonar 

7 

Poor 

SQS-26  Sonar 

7 

Poor 

Hardsite  Site  Defense 

7 

Poor 

Elevated  Kinetic  Energy  Weapon 

7 

Adequate 

Kinetic  Energy  Projectile 

7 

Adequate 

MF-30  Cannon 

1983 

Yes 

Poor 

Infrared  Search  and  Track  Designation 

System 

7 

Poor 

Anti-Aircraft  Command  and  Control 

7 

Poor 

JSTARS 

No 

Poor 

Combat  Identification  System 

7 

Poor 

Close-In  Weapon  System  (Phalanx) 

7 

Poor 

105  LGI 

7 

Poor 

RDY  Radar 

1987 

7 

Poor 

PS-05/A  Radar 

1986 

7 

Poor 

Tactical  Radar  Jamming  System 

7 

Poor 

Geometric  Arithmetic  Parallel  Processor  1987 

7 

Poor 

Airborne  Target  Handover  System 

7 

Poor 

R21G  Radar 

1985 

7 

Poor 

AN/APG-67  Radar 

1982 

7 

Poor 
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Table  A.l  (continued) 


Program  Name 

Year  First 
Operation 

Definition® 

Information 

GM-15  Countermine 

Poor 

DSP 

7 

Poor 

llOX/llOE 

7 

Poor 

Rewson 

7 

Poor 

SECS  Shipboard  Embedded  Computer  System 

7 

Poor 

CELV 

No 

Poor 

ASPS 

7 

Poor 

SUBACS 

7 

Poor 

Surface  Electronic  Warfare  System 

7 

Poor 

FATDS 

7 

Poor 

LCAC 

7 

Poor 

Patrol  Hydrofoil  Missile  Ship 

7 

Poor 

Aegis  Weapon  System 

7 

Poor 

SSN-G88  Weapon  System 

7 

Poor 

®A  question  mark  indicates  there  was  insufficient  information  to  make  a  yes  or  no 
designation. 


Table  A.2 

Programs  in  the  Program  Manager  Survey  Database 


Program  Name 

Year  First 
Operation 

Definition 

Information 

Aircraft: 

Advanced  Tactical  Fighter  (ATF) 

1990 

Yes 

Good 

AV-8B 

1978 

Yes 

Good 

C-17A 

No 

Poor 

F-14 

No 

Good 

F-16 

1974 

Yes 

Good 

OV-IOD 

1987 

Yes 

Good 

T45TS 

1989 

No 

Good 

Helicopter: 

Army  Helicopter  Improvement  Program 

(OH-58D) 

1983 

Yes 

Good 

CH-47D 

1978 

Yes 

Good 

Light  Helicopter  Experiment  (LHX) 

1993 

Yes 

Good 

V.22 

Yes 

Good 

Missile: 

Advanced  Medium  Range  Air-to-Air  Missile 

(AMRAAM) 

1981 

Yes 

Good 

FAAD  Line  of  Sight-Forward-Heavy 

1987 

Yes 

Good 
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Table  (continued) 


Program  Name 

Year  First 
Operation 

Definition 

Information 

FAAD  Non-Line  of  Sight 

1988 

Yes 

Good 

High-Speed  Anti-Radiation  Missile  (HARM) 

1987 

Yes 

Good 

Hellflre 

Yes 

Good 

IR  Maverick 

7 

Poor 

Mk  50  Tordepo 

7 

Poor 

Pershing  11 

1982 

Yes 

Good 

Sea  Lance 

1984 

Yes 

Good 

Short-Range  Attack  Missile  II  (SRAM  II) 

1985 

Yes 

Good 

TOW2B 

1988 

Yes 

Good 

Trident  11  missile 

No 

Good 

Electronic: 

Advanced  Attack  Helicopter,  Aircraft 
Survivability  Equipment  (AAH  ASE) 

1991 

Yes 

Good 

FAADC2I 

No 

Good 

GPS-User  Equipment 

1987 

Yes 

Good 

GPS-Ground  Control  Segment 

1982 

Yes 

Good 

Integrated  Defense  Avionics  Program 

1988 

Yes 

Good 

Joint  Surveillance  and  Target  Attack  System 
(JSTARS) 

No 

Poor 

Light  Airborne  Multipurpose  System  (LAMPS 

Mkm) 

7 

Poor 

MkXVIFF 

1987 

Yes 

Good 

Mobile  Subscriber  Equipment  (MSE) 

Yes 

Good 

SINCGARS 

1989 

Yes 

Good 

Submarine  Advanced  Combat  System 
(SUBACS) 

7 

Poor 

Tactical  Towed  Array  Sonar  (TACTTAS) 

1988 

Yes 

Good 

Tomahawk 

1984 

Yes 

Good 

Vehicle: 

High  Mobility  Multipurpose  Wheeled  Vehicle 
(HMMWV) 

1982 

Yes 

Good 

Ml  Block  II 

Yes 

Good 

Other: 

9mm  Pistol 

1985 

Yes 

Good 

Advanced  Attack  Helicopter,  Automated  Test 
Equipment  (AAH  ATE) 

No 

Good 

Copperhead 

Yes 

Good 

Defense  Support  Program  (DSP) 

No 

Poor 

GPS-Space  (satellite) 

1985 

Yes 

Good 
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Appendix  B 

ADDITIONAL  DETAILED  DATA  TABLES 


This  appendix  contains  tables  and  graphs  that  provide  additional 
analysis  results  for  both  the  literature  data  and  the  program  manager 
survey.  While  this  information  supports  and  informs  the  analysis 
presented  in  Sections  4  and  5,  it  was  not  included  directly  in  the  main 
text  for  reasons  of  clarity  and  presentation.  Variables  used  in  these 
tables  and  graphs  were  defined  and  discussed  in  Section  3  of  the  main 
text. 

This  appendix  has  two  sections.  This  first  presents  supplementary 
data  on  prototyping  strategies,  paralleling  the  discussion  in  Section  4. 
Many  of  these  tables  were  footnoted  in  the  main  text.  The  second 
section  presents  several  graphs  that  describe  cost  and  schedule  data 
for  both  the  literature  and  survey  data. 

SUPPLEMENTAL  DATA  ON  PROTOTYPING  STRATEGIES 

Table  B.l  shows  the  distribution  of  five  levels  of  specific  objectives 
based  on  the  program  manager  survey.  Given  that  ranking  by  impor¬ 
tance  is  implicit  here,  we  again  observe  that  objectives  relating  di¬ 
rectly  to  technological  risk  tend  to  be  more  frequent  at  higher-order 
levels  than  objectives  relating  to  operational  considerations. 

Table  B.2  illustrates  the  strong  relationships  between  main  purposes 
and  specific  objectives,  based  on  the  literature  data.  As  expected, 
technically  oriented  specific  objectives  are  more  often  associated  with 
technology-oriented  purposes.  As  the  technology  used  in  the  weapon 
system  becomes  relatively  more  mature,  operational  considerations 
begin  to  play  a  larger  role.  Table  B.3  shows  similar  results  for  the 
program  manager  survey,  but  the  relationship  here  is  less  strong. 
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Table  B.l 

Distribution  of  Specific  Objectives  (Survey  Results) 


Category 

Obj.  1 

Obj.  2 

Obj.  3 

Obj.  4 

Obj.  5 

No. 

% 

No. 

% 

No. 

% 

No. 

% 

No. 

% 

Experimental 

1 

3.2 

1 

5.6 

Exploratory 

1 

3.2 

1 

3.2 

1 

8.3 

Feasibility 

7 

22.6 

5 

16.1 

3 

11.5 

1 

8.3 

Competitive 

1 

3.2 

3 

9.7 

3 

11.5 

Developmental 

12 

38.7 

3 

9.7 

7 

26.9 

2 

16.7 

Political 

2 

7.7 

2 

11.1 

2 

16.7 

Integration 

1 

3.2 

11 

35.5 

3 

11.5 

6 

33.3 

1 

8.3 

Preproduction 

6 

19.4 

3 

9.7 

3 

11.5 

1 

5.6 

1 

8.3 

Miasionized 

2 

7.7 

1 

5.6 

1 

8.3 

Operational 

4 

12.9 

2 

7.7 

4 

22.2 

2 

16.7 

Upgrade 

2 

6.4 

1 

3.9 

1 

5.6 

1 

8.3 

Other 

1 

3.2 

2 

11.1 

Total 

31 

100 

31 

100 

26 

100 

18 

100 

12 

100 

Table  B.2 

General  Strategies  in  Prototyping  (Literature  Data) 

Objectives 

Main  Purpose 

Technology 

Viability 

Technology 

Demonstration 

System  Design 

Marketing 

I 

n 

m 

I 

n 

in 

1 

n 

m 

I 

n 

m 

Experimental 

20 

0 

0 

4 

0 

0 

0 

0 

0 

0 

0 

0 

Exploratory 

4 

17 

0 

17 

3 

0 

0 

0 

0 

0 

0 

0 

Feasibility 

1 

4 

3 

23 

18 

1 

9 

4 

0 

0 

0 

0 

Competitive 

0 

0 

1 

12 

2 

3 

13 

0 

0 

2 

0 

0 

Developmental 

0 

3 

4 

1 

18 

6 

5 

4 

0 

4 

1 

0 

Integration 

0 

0 

0 

0 

4 

2 

1 

5 

1 

0 

0 

0 

Political 

0 

0 

0 

0 

0 

2 

0 

1 

0 

0 

1 

3 

Preproduction 

0 

0 

0 

1 

1 

2 

9 

2 

3 

4 

4 

0 

Miasionized 

0 

0 

0 

0 

0 

7 

0 

9 

7 

0 

3 

0 

Operational 

0 

0 

0 

1 

1 

0 

0 

1 

2 

0 

1 

5 

Upgrade 

0 

0 

1 

0 

0 

1 

14 

0 

2 

2 

1 

0 

Total 

25 

24 

9 

59 

47 

24 

51 

26 

15 

12 

11 

8 

Table  B.3 
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Tables  B.4  through  6.6  demonstrate  relationships  between  the  three 
basic  elements  of  a  prototyping  strategy:  goals,  timing,  and  level  of 
system  integration.  Tables  6.4  and  6.5  illustrate  that  the  phase  in 
which  prototyping  occurs  is  associated  with  the  level  of  system  inte¬ 
gration.  Prototypes  that  occur  later  in  a  program  tend  to  be  more 
production  representative.  Table  6.6  shows  the  relationship  between 
the  three  elements  of  prototyping  strategy  for  the  program  manager 
survey.  The  high  frequency  of  prototyping  activities  with  system  de¬ 
sign  purposes  that  occur  during  the  FSD  phase  at  the  full  system  in¬ 
tegration  level  is  notable. 


Table  B.4 

Level  of  Integration  and  Program  Phase  (Literature  Data) 


Phase  Occurring 

Level  of  Integration 

Total 

Full  System 

Partial  System 

Subsystem 

Concept  exploration 

0 

20 

2 

22 

Demonstration/validation 

5 

13 

1 

19 

Full-scale  development 

15 

13 

8 

36 

Other  (nonprogram) 

8 

8 

4 

20 

Total 

28 

54 

15 

97 

Table  B.5 

Level  of  Integration  and  Program  Phase  (Survey  Results) 


Phase  Occurring 

Level  of  Integration 

Full  System 

Partial  System  Subsystem  Other 

Concept  exploration 

1  1 

DemonstrationAralidation 

3 

2  6 

Full-scale  development 

8 

2  4 

Production 

1 

Other 

1 

1  1 
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Table  B.6 

Elements  of  a  Prototyping  Strategy  (Survey  Results) 


Main  Purpose 

Technology 

Technology 

System 

Viability 

Demo 

Design 

Other 

Phase  occurring 

Concept  exploration 

2 

Demonstration/validation 

3 

5 

3 

Full-scale  development 
Production 

1 

1 

12 

1 

Other 

1 

2 

Level  of  integration 

Full  system 

1 

1 

10 

1 

Partial  system 

2 

3 

Subsystem 

3 

3 

6 

Other 

1 

Tables  B.7  and  B.8  demonstrate  the  wide  variation  in  prototyping 
strategies  across  weapon  system  types  and  management  agencies  for 
the  program  manager  survey,  indicating  that  at  least  at  an  aggregate 
level,  most  strategies  are  generally  applicable  across  systems  and 
management  styles. 

The  program  manager  survey  collected  information  on  contracting 
strategy.  Table  B.9  shows  the  relationship  in  contract  types  between 
phases.  While  prototyping  contracting  strategy  seems  unrelated  to 
production  contracting,  the  relationship  to  FSD  contracting  is  fairly 
strong:  Similar  contracts  appear  to  be  used  in  both  phases. 

Requirements  specification  is  another  aspect  of  contracting  strategy. 
Table  4.5  in  the  main  text  showed  this  relationship  for  the  prototype 
phase;  Table  B.IO  shows  similar  data  for  FSD  and  production  phases. 
Some  relationship  between  the  form  of  requirements  specification  and 
contract  type  can  be  seen.  During  production,  firm  fixed-price  con¬ 
tracts  contain  detailed  specifications  in  the  contract,  as  expected.  The 
FSD  phase  seems  more  varied. 

Table  B.ll  indicates  that  requirements  were  modified  as  a  result  of 
prototyping  testing  in  about  half  the  cases,  even  if  they  were  specified 
in  detail  in  the  contract.  This  result  is  not  as  counterintuitive  as  it 
first  seems:  Requirements  that  are  specified  more  loosely  do  not  need 
to  be  formally  modified  as  a  result  of  lessons  learned  throu^  proto¬ 
typing. 


Table  B.7 

Prototyping  Strategies  by  Weapon  Type  (Survey  Results) 
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System  design 

Marketing 
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Prototyping  phase 

Concept  exploration 
DemonstrationA^alidation 
Full'Scale  development 
Production 

Other 

Total 

Level  of  integration 

Full  system 

Partial  system 

Subsystem 

Other 

Total 
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Table  B.8 

Prototyping  Strategies  by  Agency  (Survey  Results) 


Management  Agency 


Air  Force 

Army 

Navy 

Joint 

Total 

Main  purpose 

Technology  viability 

1 

2 

1 

4 

Technology  demonstration 

2 

2 

2 

1 

7 

System  design 

11 

4 

4 

19 

Marketing 

Other 

1 

1 

Total 

3 

14 

8 

6 

31 

Prototyping  phase 

Concept  exploration 

1 

1 

2 

Demonstration/validation 

3 

1 

5 

2 

11 

Full-scale  development 

10 

1 

3 

14 

Production 

1 

1 

Other 

1 

1 

1 

3 

Total 

3 

14 

8 

6 

31 

Level  of  integration 

Full  system 

8 

3 

2 

13 

Partial  system 

1 

1 

1 

2 

5 

Subsystem 

2 

5 

3 

2 

12 

Other 

1 

1 

Total 

3 

14 

8 

6 

31 

Table  B.9 


Contract-Type  Tlelationships  by  Program  Phase  (Survey  Results) 


FSD  Contract 

Prototyping  Contract 

FFP 

CPI 

CPIw/c 

CPFF 

Other 

Total 

FFP 

4 

0 

0 

0 

4 

8 

CPI 

0 

6 

0 

0 

0 

6 

CPIvtr/c 

0 

0 

1 

0 

1 

2 

CPFF 

0 

0 

0 

1 

1 

2 

Other 

0 

0 

0 

0 

3 

3 

Total 

4 

6 

1 

1 

9 

21 

Production  Contract 


Prototyping  Contract 

FFP 

CPI 

CPIw/c 

CPFF 

Other 

Total 

FFP 

7 

0 

0 

0 

1 

8 

CPI 

6 

0 

0 

0 

0 

6 

CPIw/c 

2 

0 

0 

0 

0 

2 

CPFF 

1 

0 

0 

1 

0 

2 

Other 

3 

0 

0 

0 

0 

3 

Total 

19 

0 

0 

1 

1 

21 

Production  Contract 


FSD  Contract 

FFP 

CPI 

CPIw/c 

CPFF 

Other 

Total 

FFP 

4 

0 

0 

0 

0 

4 

CPI 

7 

0 

0 

0 

0 

7 

CPIw/c 

1 

0 

0 

0 

0 

1 

CPFF 

0 

0 

0 

1 

0 

1 

Other 

10 

0 

0 

0 

1 

11 

Total 

22 

0 

0 

1 

1 

24 

NOTES:  FFP  =  firm  fixed  price;  CPI  =  cost  plus  incentive;  CPIw/c  =  cost 
plus  incentive  w/ceiling;  CPFF  =  cost  plus  fixed  fee. 
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Table  B.IO 


Contract  Type  and  Requirements  Specification  (Survey  Results) 


Form  of  Requirement 

Type  of  Contract 

FFP  CPI  CPIw/c 

CPFF 

Other 

Total 

Full-Scale  Development  Phase 

Contract  specific 

2  5  1 

6 

14 

Performance  goal 

2 

1 

2 

5 

Performance  range 

1 

1 

Beat  effort 

1 

1 

Other 

2 

1 

3 

Total 

4  7  1 

1 

11 

24 

Production  Phase 

Contract  specific 

15 

15 

Performance  goal 

5 

1 

6 

Performance  range 

1 

1 

2 

Best  effort 

1 

1 

Other 

2 

2 

Total 

24 

1 

1 

26 

NOTES:  FFP  =  firm  fixed  price;  CPI  =  cost  plus  incentive;  CPIw/c  =  cost  plus  in¬ 
centive  w/ceiling;  CPFF  =  cost  plus  fixed  fee. 


Table  B.1 1 


Flexibility  to  Modify  Performance  Requirements 
(Survey  Results) 


Requirements 

Modification 

Form  of  Requirement 

Yes 

No 

Total 

Contract  specific 

11 

5 

16 

Performance  goal 

2 

5 

7 

Performance  range 

1 

2 

3 

Best  effort 

2 

0 

2 

Other 

0 

3 

3 

Total 

16 

15 

31 

Table  B.12  demonstrates  high  variation  in  contract  types  and  the 
three  basic  elements  of  prototyping  strategy. 
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Table  B.12 

Prototype  Contracting  and  Basic  Strategy  Elements 
(Survey  Results) 


Prototyping 

Contract 

Main  Purpose 

Total 

Technology 

Viability 

Technology 

Demo 

System 

Design 

Other 

FFP 

2 

4 

5 

0 

11 

CPI 

0 

3 

3 

0 

6 

CPI  w/c 

0 

0 

3 

0 

3 

CPFF 

1 

0 

2 

0 

3 

Other 

1 

1 

2 

0 

4 

Prototyping 

Contract 

Phase  in  Which  Prototyping  Occurred 

Total 

Concept 

DcmTVal. 

FSD 

Prod. 

Other 

FFP 

0 

6 

3 

0 

1 

10 

CPI 

0 

3 

3 

0 

0 

6 

CPI  w/c 

2 

0 

1 

0 

0 

3 

CPFF 

0 

1 

2 

0 

0 

3 

Other 

0 

1 

1 

0 

1 

3 

Level  of  Integration 


Prototyping 

Contract 

Full 

System 

Partial 

System 

Subsystem 

Other 

Total 

FFP 

3 

3 

4 

0 

10 

CPI 

3 

1 

2 

0 

6 

CPI  w/c 

1 

1 

1 

0 

3 

CPFF 

0 

0 

3 

0 

3 

Other 

2 

0 

1 

0 

3 

Tables  B.13  and  B.14  show  some  interesting  aspects  of  prototyping 
strategies.  For  instance,  one  possible  effect  of  prototyping  is  that  it 
may  substitute  for  all  or  some  portion  of  another  program  phase. 
Table  B.13  suggests  that  rather  than  substituting  for  another  acqui¬ 
sition  phase,  prototyping  tends  to  complement  the  traditional  phases. 
This  includes  verifying  and  validating  other  analyses  or  reducing  the 
overall  amount  of  paper  analysis.  However,  when  prototyping  is  in¬ 
tended  to  substitute  for  a  conventional  development  phase,  it  most 
likely  is  intended  as  a  streamlining  alternative.  The  Army’s  success¬ 
ful  Multiple  Launch  Rocket  System  (MLRS)  program  is  an  example  of 
this  type  of  prototyping. 


Table  B.13 


Prototype  Substitution  for  Other  Phases 


Relationship  of  Prototype  Phase 
to  Other  Phases 

Substitute? 

No  Yes 

Total 

Streamlined  alternative 

0 

1 

1 

Replaced  other  phases 

0 

3 

3 

Complementary 

7 

0 

7 

Veriiled  analysis 

3 

0 

3 

Concept/design  choice 

1 

1 

2 

Reduced  amount  of  analysis 

0 

2 

2 

Other 

2 

0 

2 

Total 

13 

7 

20 

Table  B.14 

Substitution  EBIect  and  Main  Purpose 


Relationship  of  Prototype  Phase 
to  Other  Phases 

Technology 

Viability 

Main  Purpose 

Technology 

Demo 

System 

Design 

Total 

Streamlined  alternative 

1 

1 

Replaced  other  phases 

3 

3 

Complementary 

1 

3 

3 

7 

Verified  analysis 

2 

1 

3 

Concept/design  choice 

1 

1 

2 

Reduced  amount  of  analysis 

1 

1 

2 

Other 

1 

1 

2 

Total 

3 

7 

10 

20 

Whether  prototyping  was  a  substitute  for  another  acquisition  phase 
or  was  complementary  in  one  form  or  another,  there  does  not  appear 
to  be  any  relationship  with  the  main  purpose  of  the  prototyping  effort. 
Table  B.14  shows  a  high  variability  between  the  substitution  effect 
and  purpose.  It  is  interesting  that  system  design  prototypes,  which 
are  more  often  fully  integrated  systems  built  during  FSD,  are  the  only 
kind  of  prototype  associaced  with  an  actual  substitution  effect. 

Time  trends  in  the  elements  of  protot3q)ing  strategy  based  on  the  pro¬ 
gram  manager  survey  (paralleling  Tables  4.7-4. 9  in  the  text)  are 
shown  in  Tables  B.15  through  B.17.  Interestingly,  subsystem  proto¬ 
typing  does  appear  to  be  more  common  in  more  recent  years,  while 
there  do  not  seem  to  be  strong  patterns  in  main  purposes  or  prototyp¬ 
ing  phase. 


Table  B.15 

Main  Purpoaea  over  Time  (Survey  Reaulta) 
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Main  Purpose 


Program 

Initiation 

Technology 

Viability 

Technology 

Demo 

System 

Design 

Other 

Total 

1970-1974 

1 

1 

7 

0 

9 

1976-1979 

0 

2 

2 

1 

5 

1980-1984 

2 

3 

7 

0 

12 

1985-1989 

1 

1 

2 

0 

4 

Total 

4 

7 

18 

1 

30 

NOTE:  Includes  only  programs  meeting  strict  definition;  there 
were  no  ‘marketing*  responses. 


Table  B.16 

Prototyping  Phaae  over  Time  (Survey  Results) 


Program  Initiation 

Phase  in  Which  Prototyping  Occurred 

Total 

Concept 

DemyVal. 

PSD 

Prod. 

Other 

1970  - 1974 

1 

2 

6 

0 

0 

9 

1976  -  1979 

0 

2 

1 

1 

1 

6 

1980  -  1984 

1 

5 

4 

0 

2 

12 

1985  -  1989 

0 

2 

2 

0 

0 

4 

ToUl 

2 

11 

13 

1 

3 

30 

NOTE:  Includes  only  programs  meeting  strict  deHnition. 


Table  B.17 

Level  of  Integration  over  Time  (Survey  Reaulta) 


Program 

Initiation 

Level  of  Integration 

Total 

Full  System 

Partial 

System 

Subsystem 

Other 

1970  -  1974 

5 

3 

1 

0 

9 

1976  - 1979 

4 

1 

0 

0 

5 

1980  -  1984 

3 

1 

7 

1 

12 

1985  -  1989 

1 

0 

3 

0 

4 

Total 

13 

5 

11 

1 

30 

NOTE:  Includes  only  programs  meeting  strict  definition. 
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COST  AND  SCHEDULE  DATA 

This  section  presents  some  of  the  cost  and  schedule  data  collected 
from  both  the  literature  and  the  program  manager  survey.  This  in¬ 
formation  is  presented  here  rather  than  in  the  main  text,  as  it  is  not 
directly  relevant  to  the  main  arguments  of  this  research.  In  general, 
relationships  between  cost,  schedule,  and  prototyping  are  fairly  weak 
and  are  in  the  directions  we  might  expect  based  on  other  research. 
No  strong  implications  for  prototyping  strategies  can  be  drawn.  How¬ 
ever,  cost  and  schedule  estimates  are  important  parts  of  any  prototyp¬ 
ing  strategy,  and  cost  and  schedule  outcomes  are  one  (indirect)  way  of 
measuring  the  relative  success  or  failure  of  that  strategy. 

Figure  B.l  shows  the  costs  of  prototyping  and  total  development  (in 
constant  FY82  dollars)  for  the  literature  data  as  a  function  of  the  year 
of  program  initiation,  measured  as  the  Milestone  I  (or  equivalent) 
date.  No  strong  pattern  emerges.  Consistent  with  prior  research, 
prototyping  appears  to  remain  a  relatively  small  part  of  development 
costs.  Figure  B.2  shows  just  the  prototype  costs;  with  a  few  excep¬ 
tions,  most  prototypes  in  this  data  set  cost  under  $400  million 
(FY82$). 

Figure  B.3  shows  similar  data  for  the  programs  in  the  program  man¬ 
ager  survey  (in  constant  FY89  dollars).  The  same  basic  distribution  is 
apparent.  Figures  B.4  and  B.5  show  the  cost  of  the  prototype  as  a 
percent  of  development  costs  and  total  program  costs  respectively. 
While  the  cost  of  prototyping  can  be  a  large  part  of  development  cost, 
it  is  generally  a  fairly  small  portion  of  total  program  costs. 

Schedule  data  are  shown  in  Figure  B.6  for  the  literature  database. 
Design  time  is  measured  as  the  months  between  program  start  and 
first  prototype  operation.  The  length  of  the  prototyping  phase  in¬ 
cludes  design  time,  but  extends  to  the  end  of  the  prototyping  test 
phase  when  (hopefully)  the  objectives  have  been  achieved.  Total  pro¬ 
gram  duration  is  measured  as  the  time  between  program  start 
(Milestone  I  or  equivalent)  and  first  operational  delivery.  Figure  B.7 
shows  that  the  length  of  the  prototyping  phase  as  a  percent  of  total 
program  duration  has  been  decreasing. 

Figures  B.8  through  B.ll  provide  some  detail  on  prototype-related 
schedule  trends  using  the  program  manager  survey.  The  length  of 
the  prototyping  phase  (Figure  B.8,  measured  as  above)  and  the  proto¬ 
type  design  time  (Figure  B.9,  measured  as  above)  have  varied  consid¬ 
erably  over  time,  but  no  strong  pattern  emerges.  On  the  other  hand, 
both  the  time  from  progpram  start  to  first  operation  (Figure  B.IO)  and 
the  time  to  when  prototyping  objectives  have  been  achieved  (Figure 
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B.ll)  have  decreased.  Reasons  for  that  change  are  not  currently 
known. 
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Figure  B.l — Cost  Trends  over  Time  (Literature  Data) 
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Figure  B.2 — Prototype  Cost  Trends  (Literature  Data) 
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Figure  B.5— Prototype  Cost  as  a  Percent  of  Total  Cost 
(Program  Manager  Survey) 
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Figure  B.6— Schedule  Trends  (Literature  Data) 
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Figure  B.7 — Prototype  Phase  as  a  Percent  of  Program  Length 
(Literature  Data) 
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Figure  B.8 — Length  of  Prototyping  Phase 
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Figure  B.9 — Prototype  Design  Time 
(Program  Manager  Survey) 


1971  1973  1975  1977  1979  1981  1983  1985  1987 

Year  of  program  initiation 

Figure  B.IO — Time  to  First  Prototype  Operation 
(Program  Manager  Survey) 
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Appendix  C 

PROTOTYPING  STRATEGY  WORKSHEET 


A  mcgor  part  of  the  research  design  for  this  analysis  was  the  design  of 
a  prototyping  strategy  worksheet  as  a  mechanism  for  collecting  in¬ 
formation  on  current  prototyping  activities.  The  worksheet  is  pre¬ 
sented  here.  It  was  sent  to  the  government  program  managers  of  85 
current  U.S.  weapon  system  programs:  43  responses  were  received 
covering  41  programs  (listed  in  Table  A.2).  In  most  cases,  there  was 
at  least  one  follow-up  phone  conversation  after  receipt  of  the  work¬ 
sheet. 

The  worksheet  consists  of  four  sets  of  questions:  general  information, 
management  issues,  cost  and  schedule,  and  technology  and  perfor¬ 
mance.  Most  of  the  questions  are  qualitative  in  nature  and  the  an¬ 
swers  were  therefore  subjective.  The  focus  of  the  worksheet  is  on  the 
role  of  the  prototype  or  prototyping  phase  in  the  program  and  the 
usefulness  of  the  prototyping  activities  with  respect  to  achievement  of 
program  objectives. 

At  the  time  the  survey  was  implemented,  data  of  this  sort  had  never 
before  been  collected  and  analyzed  for  this  many  programs. 


% 
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3/18/88 


PROTOTYPING  STRATEGY  WORKSHEET 


PERSON  COMPLETING  WORKSHEET: 


Name:  _ 

Title:  _ 

#  Years  at  Position: 
Phone  Number: 

DATE  COMPLETED: 


Recently  there  has  been  increased  interest  in  the  use  of  prototyping  during  weapon  system 
development.  Prototyping  held  a  prominent  place  in  the  Packard  Commission  report, 
Congress  is  now  encouraging  the  use  of  prototypes,  and  prototyping  is  now  formally  a  part 
of  DoD  regulations  governing  major  weapon  system  development  (DoDI  5000.2). 
However,  there  remain  questions  such  as  what  a  prototype  is,  under  what  conditions  it 
makes  sense  to  prototype  and  what  the  benefits  of  prototyping  might  be.  The  Office  of  the 
Undersecretary  of  Defense  (Acquisition)  is  sponsoring  a  RAND  Corporation  study  of 
prototyping  activities  which  begins  to  address  these  questions. 

The  purpose  of  this  worksheet  is  to  collect  previously  unavailable  information  from  various 
weapon  system  development  programs.  These  data  will  be  aggregated  to  test  hypotheses 
regarding  the  uses  of  prototypes  and  the  relationship  between  prototyping  activities  and  the 
acquisition  environment. 

Though  a  considerable  effort  has  been  made  to  limit  the  amount  of  information  that  is 
needed,  we  recognize  that  the  requested  information  is  rather  extensive.  We  have, 
therefore,  designed  the  questions  to  be  mostly  subjective  in  nature.  The  worksheet  should 
be  completed  by  someone  in  the  Program  Office  closely  connected  with  the  program  (e.g., 
program  manager  or  deputy,  or  chief  engineer).  All  information  that  would  allow 
identiFication  of  a  specific  program  will  be  treated  as  confidential  by  RAND.  Please  feel  free 
to  call  your  RAND  contact  (collect)  if  you  have  any  questions. 


RAND  CONTACT:  Jeff  Drezner 
TELEPHONE:  (213)393-0411 


i 


Prototype. 


Prototype  Phase: 

Other  Phases. 

Program  Initiation; 

Demonstration/ 

Validation: 

Full-Scale 

Development: 

Low-Rate 
Initial  Production: 

Full  Production. 
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GLOSSARY 


An  article  fabricated  in  the  expectation  of  change,  often  in  the 
demonstration  validation  phase.  Prototypes  differ  from  full-scale 
development  articles  in  terms  of  their  goals  and  the  subsequent 
decisions  affected.  This  implies  that  the  results  of  the  prototype 
phase  will  be  the  basis  for  one  or  more  major  decisions,  such  as 
cost,  schedule,  and  performance  trade-offs,  or  program  termi¬ 
nation,  prior  to  commitment  to  full-scale  production.  For  any  given 
program,  there  may  be  one  or  more  of  several  different  kinds  of 
prototypes. 

Phase  of  the  acquisition  process  concerned  with  design, 
fabrication,  and  testing  of  prototypes.  May  be  a  portion  of  the  time 
or  the  entire  time  between  major  decision  milestones,  or  it  may  cut 
across  decision  milestones.  This  phase  often  corresponds  with  the 
demonstration  validation  phase. 

Phase(s)  of  the  acquisition  process  not  directly  concerned  with 
prototyping:  may  include  all  or  part  of  concept  exploration, 
demonstration/validation,  full-scale  development,  low-rate  initial 
production,  full-scale  production. 

Formal  point  in  time  at  which  program  was  begun.  May  coincide 
with  Milestone  "O’  or  I,  establishment  of  Program  Office,  or  some 
other  event. 


Phase  I  or  equivalent. 

Phase  II  or  equivalent. 

Milestone  Ilia  or  equivalent. 
Milestone  III  or  lllb  decision 
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I.  GENERAL  INFORMATION 


Program  Name:  _ 

1 .  Weapon/System  Ty  pe : 

(Check  One) 

□  Fixed-Wing  Aircraft 

□  Helicopter 

□  Missile,  Bomb,  Torpedo 

□  Communications,  Navigation 

□  Other- Specify:  _ 

2.  Weapon/System  Function  or  Mission: 

3.  Management  Agency  . 

(Check  One) 

□  Air  Force 

□  Army 

□  Navy/Marine  Corps 

□  Joint  -  Specify: _ 


4.  Does  the  term  'prototype'  have  a  specific  meaning  in  the  context  of  your  program? 
Please  explain. 


Do  you  distinguish  between  prototypes  and  other  kinds  of  preproduction  articles  in 
your  program?  Please  explain. 


What  was  (is)  the  overall  purpose  of  prototyping  in  the  program:  What  were  (are) 
the  expect^  benefits  of  including  a  prototype  phase? 


Describe  the  effect  of  the  prototype  phase  on  the  overall  program.  The  interest 
here  is  on  how  the  information  generated  during  the  prototype  phase  was  (will  be) 
used.  For  example,  were  (will)  cost  and  schedule  estimates  (be)  improved  or 
adjusted  based  on  the  results  of  the  prototype  phase?  Did  (will)  information  from 
the  prototype  cause  performance  requirements  to  be  adjusted? 
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8.  For  each  of  the  following  levels,  indicate  if  there  was  a  prototype  and  if  so,  the 
number  of  prototypes  built.  For  example,  an  engine/platform  combination  counts  as 
a  partial  system  prototype,  and  adding  fully  integrated  electronics  to  the  powered 
platform  produces  a  full  system.  The  definitions  for  each  level  of  prototyping  are 
provided  below. 

Level  of  Prototyping  ^  Yes  Number 


Full  System .  □  □  — >  How  many? 

Partial  System .  □  □  — >  How  many? 

Subsystem  .  □  □  — >  How  many? 

Component  .  □  □  — >  How  many? 


Other  -  Explain 


Definitions 

Full  System:  Fully  integrated  unit,  with  all  key  subsystems  included,  in  final 

or  close  to  final  configuration. 

Partial  System:  Less  than  fully  integrated  system,  with  some  key  subsystems 

included,  not  necessarily  fin^  form. 

Subsystem:  Key  parts  of  weapon  system  which  are  ‘stand  alone’  (e  g., 

platform,  propulsion,  various  types  of  electronics,  gun  system). 

Components:  Building  blocks  of  subsystems  (e.g,.  Very  High  Speed 

Integrated  Circuits  VHSIC). 


9.  Many  of  the  remaining  questions  in  the  worksheet  should  be  answered  with  a  single 
prototype  article  in  mind.  Please  indicate  which  level  of  prototyping  (see  Question 
8)  you  chose  and  within  that  level  which  prototype  if  there  is  more  than  one  at  that 
level.  Briefly  explain  why  you  chose  that  prototype.  Our  preference  is  for  the  one 
you  believe  contributes  the  most  to  achieving  the  objectives  of  the  program. 
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10.  We  would  like  to  know  the  purpose  or  use  of  the  prototype  you  have  chosen  (see 
Question  9).  Attachment  A  is  a  taxonomy  we  have  developed  for  classifying 
prototypes.  Using  this  taxonomy,  we  would  like  you  to  classify  the  prototype.  In  the 
boxes  below,  first  indicate  which  of  the  general  purposes  apply,  by  entering  the 
corresponding  letter,  and  note  secondary  purposes  if  any.  Next,  indicate  the  specific 
objectives  associated  with  the  prototype  by  entering  the  corresponding  numbers  in 
decreasing  order  of  importance.  If  the  categories  described  do  not  capture  the 
objectives  adequately,  please  use  your  own  term  to  describe  the  objective. 


Main  Purposes 

Specific  Objectives 

A. 

Uncertainty  Reduction 

1. 

Experimental 

B. 

Technology  Demonstration 

2. 

Exploratory 

C. 

System  Design/PerformanceValidation 

3. 

Feasibility 

D. 

Marketing 

4. 

Competitive 

O. 

Other  -  Specify: 

5 

Developmental 

6. 

Political 

7. 

Integration 

8. 

Preproduction 

9 

Missionized 

10. 

Operational 

11. 

Upgrade 

12 

Other 

Specify: 


Secondary 
Main  Purpose 

Purpose  (If  exists) 


Specific  objectives  in  decreasing 
order  of  importance 


II.  MANAGEMENT 

Questions  1 1  through  13  are  about  the  prototype  you  selected  in  Question  9. 

1 1 .  How  were  (will)  the  performance  requirements  (be)  specified  for  the  prototype? 
(Check  One) 

□  Specific  Contract  Requirements 

□  Performance  Goal 

□  Performance  Range 

□  Best  Efforts 

□  Other  -  Specify: _ 


12.  Were  (Will)  the  perlormance  ^uirements  for  the  final  system  (be)  modified  as  a 
result  of  prototype  testing?  This  includes  reliability  and  maintainability  requirements. 
Please  explain. 


13.  Indicate  the  type  of  contract  used  for  the  prototype  phase  and  then  for  the  full-scale 


development  and  procurement  phases  of  ^e  program. 

Prototype  ESQ 

(Check  One  In  Each 

Procurement 

Column) 

Firm  Fixed  Price 

□ 

□ 

□ 

Cost  *  Incentive 

□ 

□ 

□ 

Cost  *  Incentive  with  ceiling 

□ 

□ 

□ 

Cost  +  Fixed  Fee 

□ 

□ 

□ 

Other  -  Specify: 

□ 

□ 

□ 

1 4.  Did  (Will)  the  prototype  and  associated  testing  substitute  for,  or  provide  an  alternative 
to,  other  kinds  of  analysis  or  development  steps?  Please  explain. 


H3 


Questions  15-19  refer  to  the  entire  program. 

15.  As  compared  to  a  similar  nonprototype  development  program,  how  would  you  rate 
the  amount  of  documentation  and  reporting  required  in  the  prototype  phase  of  the 
program? 

(Circle  One  Number) 

1  2  3  4  5 

Less  Same  More 

16.  Beginning  with  program  initiation,  for  each  year  of  the  program,  what  was  the 
average  professional  staff  size  (number  of  full-time  equivalents)  in  the  system 
program  office? 

Fiscal  Year  I  Fiscal  Year  f  Fiscal  Year  K 


1 7.  Indicate  the  number  of  organizational  layers  between  the  Project  Manager  and  the 
person  who  can  authorize  major  program  changes  during  the  prototype  phase,  full- 
scale  development  and  during  production.  A  major  program  change  might  constitute 
a  reduction  or  increase  in  performance,  the  addition  or  subtraction  of  a  mission 
requirement,  schedule  stretches,  or  cost  increases. 

#  Layers  During  Prototype  Phase  _ 

#  Layers  During  Full-Scale  Development  _ 

#  Layers  During  Production  Phase  _ 

18.  Indicate  who  (position,  not  name)  the  top  decision  maker  was  (will  be)  for  each  of  the 
following  types  of  changes. 

Cost  changes  _ 

Schedule  changes  _ 

Performance  changes  _ 

19.  Was  there  (Will  there  be)  industry  teaming  on  this  project? 

(Check  One) 

□  Yes 


□ 


No 
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III.  COST  AND  SCHEDULE 


20.  Indicate  the  cost  to  the  government  (actual  or  most  current  estimate)  in  constant 
dollars  for  the  following.  The  cost  of  the  prototype  phase  is  a  subset  of  RDT&E 
expenditures  and  includes  engineering,  tooling,  hardware  and  testing  costs 
associated  with  the  prototype.  Also,  indicate  the  quantity  associated  with  the  cost 
figure  provided.  Of  interest  here  is  the  cost  of  the  prototype  phase  as  a  percent  of 


total  RDT&E  and  procurement  costs. 

Dollars 

Year  Dollars  In 

Quantity 

Research,  Development, 
Test  &  Evaluation 

$ 

19 

Prototype 
(all  prototypes) 

S 

19 

System  Procurement 

$ 

19 

21.  Indicate  the  actual  or  most  current  schedule  estimates  for  the  following  program 
milestone  dates; 


Program  Initiation 

m/m 

MO  YR 

DemonstrationAfalidation 

m/m 

(Milestone  1) 

MO  YR 

Full-Scale  Development 

m/m 

(Milestone  II) 

MO  YR 

Low-Rate  Initial  Production 

m/m 

(Milestone  Ilia) 

MO  YR 

Full  Production 

m/m 

(Milestone  lllb) 

MO  YR 

Specify  Event 


22.  For  the  prototype  selected  indicate  the  actual  or  most  current  estimate  of  dates  that 
the  following  occured  or  will  occur.  Objectives  achieved  refers  to  the  date  at  which 
information  generated  by  prototype  testing  was  sufficient  to  move  on  to  the  next 
phase  in  the  program. 


Design 

Sian 

□□/m 

MO  YR 


Fabrication 

Start 

m/m 

MO  YR 


First 

Operation 

m/m 

MO  YR 


Objectives 

Achieved 

m/m 

MO  YR 
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23.  Please  indicate  the  start  and  completion  dates  for  the  entire  prototype  phase. 

START  COMPLETION  Co/m 

Month  Year  Month  Year 

24.  When  was  the  decision  made  to  prototype? 

ENTER  DATE  □□/ CD 

Month  Year 

24a.  Who  made  this  decision  (position)? _ 

25.  Was  the  prototype  phase  planned  from  the  outset? 

(Check  One) 

□  Yes 

□  No 

IV.  TECHNOLOGY  AND  PERFORMANCE 

26a.  Was  (Is)  the  advance  sought  in  the  overall  program  evolutionary  (relatively  small 
increase  in  technological  advance  building  on  the  existing  state  of  the  art  as 
represented  by  existing  systems)  or  revolutionary  (major  innovative  technological 
advance  over  current  systems). 

(Check  One) 

□  Evolutionary 

□  Revolutionary 

26b.  What  system  did  you  use  as  a  basis  for  deciding  whether  this  program  was 
evolutionary  or  revolutionary? 


27.  What  were  (will  be)  the  most  difficult  technical  challenges  in  the  overall  program? 
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28.  Describe  the  role  of  the  prototype  phase  in  meeting  these  challenges. 


29.  Describe  the  fabrication  methods  used  (to  be  used)  to  build  the  prototype  selected. 
Include  whether  CAD/CAM  was  (will  be)  used  and  how  prototype  fabrication  differs 
from  production  fabrication  (e  g  ,  same  or  different  plant,  tooling,  etc.). 


30.  How  would  you  rate  the  skill  level  of  the  contractor  personnel  involved  in  the 
prototype  phase  compared  to  those  used  by  the  contractor  in  the  rest  of  the 
program?  For  example,  it  is  sometimes  the  case  that  smaller,  more  highly  skilled 
design  teams  are  used  by  the  contractor  for  prototype  design,  fabrication,  and 
testing. 

(Cirde  One  Number) 

1  2  3  4  5 

Less  Same  More 

Skilled  Skilled 

31 .  What  additional  information  do  you  feel  would  be  needed  to  help  us  fully  understand 
the  prototyping  activities  in  your  program? 
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Attachment  A 


There  are  four  general  purpose  categories:  uncertainty  reduction,  technology 
demonstration,  design/performance,  and  marketing.  These  categories  represent  the 
overall  purpose  or  use  of  the  prototype  and  are  dosely  related  to  the  expected  benefits  of 
the  prototype  and  the  decision  stage  of  the  program. 

1.  Uncertainty  Reduction: 

This  purpose  is  oriented  toward  generating  information  to  reduce  technological  risk  in  a 
general  sense.  These  are  'building  block'  prototypes,  meaning  that  they  add  to  the 
general  knowledge  base.  These  prototypes  generally  occur  early  in  a  program  (before 
demonstration/validation  at  Milestone  I).  No  military  mission  needs  to  be  specified. 

2.  Technology  Demonstration: 

This  purpose  concerns  exploring  the  possibte  performance  envelope  of  a  system.  The 
prototypes  in  this  category  are  often  used  for  exploring  the  usefulness  of  a  new  design 
or  concept  to  meet  a  mission,  and  demonstrations  or  a  particular  application  of 
technology.  Prototypes  in  this  category  are  also  used  to  generate  or  preserve  design 
and  concept  options  as  a  hedge  against  threat  uncertainty.  Missions  or  functional 
requirements  are  often  specified.  These  prototypes  may  occur  early  in  the  program  in 
concept  exploration  or  validation  phases  at  a  time  when  the  design  is  not  frozen. 
Production  is  often  anticipated. 

3.  System  Design/Performance  Validation: 

This  purpose  relates  to  prototype  uses  involving  design/performance  specifications  or 
requirements.  Also  included  here  are  demonstrations  of  the  ability  to  meet  a  specified 
threat,  contract  specifications,  and  producibility  concerns.  Missions  are  specified,  often 
in  detail,  and  there  is  an  expectation  of  production.  Operation,  support,  and  logistics 
are  also  of  concern.  This  category  might  also  be  called  'engineering'  as  these 
prototypes  are  often  fabricated  as  part  of  advanced  development  or  full-scale 
engineering  effort. 

4.  Marketing: 

This  category  refers  to  uses  having  to  do  directly  with  selling  a  product  or  supporting  a 
proposal.  These  are  often  close  to  production  configuration  or  missionized. 
Competition  is  a  frequent  use  here,  or  some  other  variation  on  the  theme  that  'mine  is 
better  than  yours.'  These  prototypes  can  be  part  of  any  decision  phase  prior  to 
production,  and  there  is  a  definite  expectation  tor  production.  Missions  do  not  need  to 
be  specified,  though  the  prototype  is  always  oriented  toward  a  specific  functional 
requirement.  These  prototypes  are  sometimes  funded  by  private  industry. 


Specific  objectives  are  the  next  level  of  purpose  below  the  general  purpose.  This  level 
explicitly  recognizes  that  there  are  many  possible  specific  uses  for  prototypes,  and  that 
several  olqectives  might  be  appropriate  for  one  prototype.  Specific  objectives  may  also  cut 
across  the  boundries  of  the  main  purpose  categories  and  relate  to  the  SF>ecific  uses  a 
prototype  was  built  tor,  and/or  to  the  specific  information  generated.  There  are  1 1  specific 
objectives  listed  and  defined  below. 

•  Experimental:  Demonstrates  a  new  idea,  new  technology,  or  existing  technology 

in  a  new  application.  Usually  occurs  very  early  in  the  program 
and  may  not  have  a  particular  mission  or  expectations  of 
production. 


Exploratory; 


Feasibility: 


Competitive; 

Developmental: 


Political: 


Integration; 


Preproduction: 

Missionized; 


Operational: 


Upgrade. 


Evaluates  possible  performance  envelope  or  tests  feasibility  of 
performance  requirement.  May  not  have  mission  specified  or 
expectations  of  production,  but  does  have  explicit  performance 
goals.  Usually  occurs  in  oorrcept  or  validation  phase. 

Demonstrates  performance  objectives  in  reference  to  a  specific 
mission.  Usually  occurs  in  validation  phase,  though  production 
may  not  be  expected. 

Used  to  improve  source-selection  decisions  in  validation  or  PSD 
phases.  Production  is  anticipated. 

Determines  the  tactical  or  operational  suitability  for  military  uses. 
May  occur  in  concept  or  validation  phases.  This  is  the 
missionized  version  of  an  experimental  prototype. 

Achieves  some  political  or  corporate  strategy  objective,  or 
demonstrates  attainment  of  a  political  objective,  or  responds  to  a 
politically  established  requirement.  Can  occur  throughout  the 
decision  process,  though  it  occurs  most  often  in  validation  or 
FSD. 

Tests  subsystem  matching  and  full  system  operation.  May  be 
part  of  concept,  validation,  or  FSD  pluses.  Specific  mission  or 
functional  requirement  exists. 

Tests  production  configuration,  after  design  freeze,  usually  during 
FSD.  Full  rate  production  is  expected. 

Evaluates  performance  with  respect  to  specified  threat  using  fully 
integrated  system.  May  occur  In  concept,  validation,  or  FSD 
phases. 

Tests  operational  suitability  of  fully  integrated  system,  including 
reliability,  availability,  and  maintainability  characteristics.  Also 
used  for  doctrine  development  and  integrated  logistics  support 
planning. 

Tests  or  demonstrates  subsystem  improvement  to  existing 
system  in  operational  use.  Occurs  during  production  phase  of 
existing  platform,  but  the  upgrade  itself  may  be  part  of  a  concept, 
validation,  or  FSD  phase. 
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